HOUSE PRICE PREDICTION¶
Structure:¶
- Introduction
- Data Loading
- EDA - Univariate
- EDA - Bivariate
- Data Preprocessing
- Model Building with Dataset-1
- Hypertuning Dataset-1
- Summary - Dataset-1
- Model Building with Dataset-2
- Hypertuning Dataset-2
- Summary - Dataset -2
- Conclusion
- Pickle file creation
Note:
Dataset - 1 = 22 features
['price', 'room_bed', 'room_bath', 'living_measure', 'lot_measure',
'ceil', 'coast', 'sight', 'condition', 'quality', 'ceil_measure',
'basement', 'yr_built', 'living_measure15', 'lot_measure15',
'furnished', 'total_area', 'month_year', 'City', 'has_basement',
'HouseLandRatio', 'has_renovated']
Dataset - 2 = 31 features (important features after imputing dummy and analyzing different models)
['price', 'room_bed', 'room_bath', 'living_measure', 'lot_measure',
'ceil', 'sight', 'condition', 'ceil_measure', 'basement', 'yr_built',
'yr_renovated', 'zipcode', 'lat', 'long', 'living_measure15',
'lot_measure15', 'total_area', 'coast_1', 'quality_3', 'quality_4',
'quality_5', 'quality_6', 'quality_7', 'quality_8', 'quality_9',
'quality_10', 'quality_11', 'quality_12', 'quality_13', 'furnished_1'
Prerequisites for the running the file:¶
Below are 2 files needed to be added to you current working directory.
1. Need to add file USA ZipCodes_1.xlsx to current working directory to access this data
2. Add the folder WA to your current working directory
3. Install below 2 libraries
conda install -c conda-forge/label/cf201901 geopandas
conda install -c conda-forge/label/cf201901 shapely
</b>
This Jupyter Notebook is done as part of PGPML Great Learning Programme for Capstone Project. Let's first, define the problem, objective of this excercise.
We have the problem statment well defined in the given document which is as follows
INTRODUCTION¶
Problem Statement¶
As a house value is simply more than location and square footage. Like the features that make up a person, an educated party would want to know all aspects that give a house its value. For example, if we want to sell a house and we don't know the price which we can take, as it can't be too low or too high. To find house price we usually try to find similar properties in our neighbourhood and based on collected data we trying to assess our house price.
Problem Definition¶
When any person/business wants to sell or buy a house, they always face this kind of issue as they don't know the price which they should offer. Due to this they might be offering too low or high for the property. Therefore, we can analyze the available data of the properties in the area and can predict the price. We need to find how these attributes influence the house prices Right pricing is very imporatnt aspect to sell house. It is very important to understand what are the factors and how they influence the house price. Objective is to predict the right price of the house based on the attributes
Objective¶
Build model which will predict the house price when required features passed to the model. So we will
Business Reason¶
As people don't know the features/aspects which commulate property price, we can provide them HouseBuyingSelling guiding services in the area so they can buy or sell their property with most suitable price tag and they didn't lose their hard earned money by offering low price or keep waiting for buyers by putting high prices.
DATA LOADING¶
First, we will load the data from the given csv(comma seperated values) file provided as part of the Capstone Project.
In [2]:
# loading the library required for data loading and processing
import pandas as pd
import numpy as np
#Supress warnings
import warnings
warnings.filterwarnings('ignore')
# read the data using pandas function from 'innercity.csv' file
house_df = pd.read_csv('innercity.csv')
In [3]:
# let's check whether data loaded successfully or not, by checking first few records
house_df.head()
Out[3]:
Data is loaded successfully as we can see first 5 records from the dataset.
Data Understanding¶
After loading data into our pandas library dataframe, we can now try to understand the kind of data we have with us.
In [4]:
# print the number of records and features/aspects we have in the provided file
house_df.shape
Out[4]:
We have more than 21k records having 23 features
In [5]:
# let's check out the columns/features we have in the dataset
house_df.columns
Out[5]:
From the above we can see the different columns we have in dataset.
These columns provide below information
- cid: Notation for a house. Will not of our use. So we will drop this column
- dayhours: Represents Date, when house was sold.
- price: It's our TARGET feature, that we have to predict based on other featues
- room_bed: Represents number of bedrooms in a house
- room_bath: Represents number of bathrooms
- living_measure: Represents square footage of house
- lot_measure: Represents square footage of lot
- ceil: Represents number of floors in house
- coast: Represents whether house has waterfront view. It seems to be a categorical variable. We will see in our further data analysis
- sight: Represents how many times sight has been viewed.
- condition: Represents the overall condition of the house. It's kind of rating given to the house.
- quality: Represents grade given to the house based on grading system
- ceil_measure: Represents square footage of house apart from basement
- basement: Represents square footage of basement
- yr_built: Represents the year when house was built
- yr_renovated: Represents the year when house was last renovated
- zipcode: Represents zipcode as name implies
- lat: Represents Lattitude co-ordniates
- long: Represents Longitude co-ordinates
- living_measure15: Represents square footage of house, when measured in 2015 year as house area may or may not changed after renovation if any happened
- lot_measure15: Represents square footage of lot, when measured in 2015 year as lot area may or may not change after renovation if any done
- furnished: Tells whether house is furnished or not. It seems to be categorical variable as description implies
- total_area: Represents total area i.e. area of both living and lot
In [6]:
# let's see the data types of the features
house_df.info()
In the dataset, we have more than 21k records and 23 columns, out of which
In [7]:
# let's check whether our dataset have any null/missing values
house_df.isnull().sum()
Out[7]:
We don't have any null or missing values for any of the columns
In [8]:
# let's check whether there's any duplicate record in our dataset or not. If present, we have to remove them
house_df.duplicated().sum()
Out[8]:
We don't have any duplicate record in out dataset. So we can say we have more than 21k Unique records
In [9]:
# let's do the 5 - factor analysis of the features
house_df.describe().transpose()
Out[9]:
- CID: House ID/Property ID.Not used for analysis
- Dayhours: 5 factor analysis is reflecting for this column
- price: Our taget column value is in 75k - 7700k range. As Mean > Median, it's Right-Skewed.
- room_bed: Number of bedrooms range from 0 - 33. As Mean slightly > Median, it's slightly Right-Skewed.
- room_bath: Number of bathrooms range from 0 - 8. As Mean slightly < Median, it's slightly Left-Skewed.
- living_measure: Square footage of house range from 290 - 13,540. As Mean > Median, it's Right-Skewed.
- lot_measure: Square footage of lot range from 520 - 16,51,359. As Mean almost double of Median, it's Hightly Right-Skewed.
- ceil: Number of floors range from 1 - 3.5 As Mean ~ Median, it's almost Normal Distributed.
- coast: As this value represent whether house has waterfront view or not. It's categorical column. From above analysis we got know, very few houses has waterfront view.
- sight: Value ranges from 0 - 4. As Mean > Median, it's Right-Skewed
- condition: Represents rating of house which ranges from 1 - 5. As Mean > Median, it's Right-Skewed
- quality: Representign grade given to house which range from 1 - 13. As Mean > Median, it's Right-Skewed.
- ceil_measure: Square footage of house apart from basement ranges in 290 - 9,410. As Mean > Median, it's Right-Skewed.
- basement: Square footage house basement ranges in 0 - 4,820. As Mean highlty > Median, it's Highly Right-Skewed.
- yr_built: House built year ranges from 1900 - 2015. As Mean < Median, it's Left-Skewed.
- yr_renovated: House renovation year only 2015. So this column can be used as Categorical Variable for knowing whether house is renovated or not.
- zipcode: House ZipCode ranges from 98001 - 98199. As Mean > Median, it's Right-Skewed.
- lat: Lattitude ranges from 47.1559 - 47.7776 As Mean < Median, it's Left-Skewed.
- long: Longittude ranges from -122.5190 to -121.315 As Mean > Median, it's Right-Skewed.
- living_measure15: Value ragnes from 399 to 6,210. As Mean > Median, it's Right-Skewed.
- lot_measure15: Value ragnes from 651 to 8,71,200. As Mean highly > Median, it's Highly Right-Skewed.
- furnished: Representing whether house is furnished or not. It's a Categorical Variable
- total_area Total area of house ranges from 1,423 to 16,52,659. As Mean is almost double of Median, it's Highly Right-Skewed
From above analysis we got to know,
Most columns distribution is Right-Skewed and only few features are Left-Skewed (like room_bath, yr_built, lat).
We have columns which are Categorical in nature are -> coast, yr_renovated, furnished
Exploratory Data Analysis¶
Let's do some visual data analysis of the features
Univariate Analysis - By BoxPlot¶
In [10]:
#let's first import the required libraries for the plots
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
# size of plots to make it uniform throughout our analysis in the notebook
plotSizeX = 12
plotSizeY = 6
# let's boxplot all the numerical columns and see if there any outliers
for i in house_df.iloc[:, 2:].columns:
house_df.iloc[:, 1:].boxplot(column=i)
plt.show()
We can see, there are lot of features which have outliers. So we might need to treat those before building model
Analyzing Feature: cid¶
In [11]:
#cid - CID is appearing muliple times, it seems data contains house which is sold multiple times
cid_count=house_df.cid.value_counts()
cid_count[cid_count>1].shape
Out[11]:
We have 176 properties that were sold more than once in the given data
Analyzing Feature: dayhours¶
In [12]:
#we will create new data frame that can be used for modeling
#We will convert the dayhours to 'month_year' as sale month-year is relevant for analysis
house_dfr=house_df.copy()
house_df.dayhours=house_df.dayhours.str.replace('T000000', "")
house_df.dayhours=pd.to_datetime(house_df.dayhours,format='%Y%m%d')
house_df['month_year']=house_df['dayhours'].apply(lambda x: x.strftime('%B-%Y'))
house_df['month_year'].head()
Out[12]:
We successfully converted dayhours feature to month_year for better analysis.
In [13]:
house_df['month_year'].value_counts()
Out[13]:
We can see, most houses sold in April, July month
In [14]:
house_df.groupby(['month_year'])['price'].agg('mean')
Out[14]:
So the time line of the sale data of the properties is from May-2014 to May-2015 and April month have the highest mean price.
Analyzing Feature: Price (our Target)¶
In [15]:
house_df.price.describe()
Out[15]:
In [16]:
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.distplot(house_df['price'])
Out[16]:
The Price is ranging from 75,000 to 77,00,000 and distribution is right-skewed.
Analyzing Feature: room_bed¶
In [17]:
house_df['room_bed'].value_counts()
Out[17]:
The value of 33 seems to be outlier we need to check the data point before imputing the same
In [18]:
house_df[house_df['room_bed']==33]
Out[18]:
Will delete this data point after bivariate analysis as it looks to be an outlier as it has low price for 33 bed room property
In [19]:
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.countplot(house_df.room_bed,color='green')
Out[19]:
Most of the houses/properties have 3 or 4 bedrooms
Analyzing Feature: room_bath¶
In [20]:
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.countplot(house_df.room_bath,color='green')
house_df['room_bath'].value_counts().sort_index()
Out[20]:
Majority of the properties have bathroom in the range of 1.0 to 2.5
In [21]:
plt.figure(figsize=(plotSizeX, plotSizeY))
print("Skewness is :",house_df.room_bath.skew())
sns.distplot(house_df.room_bath)
Out[21]:
Analyzing Feature: Living measure¶
In [22]:
#Data is skewed as visible from plot, as its distribution is normal
plt.figure(figsize=(plotSizeX, plotSizeY))
print("Skewness is :",house_df.living_measure.skew())
sns.distplot(house_df.living_measure)
house_df.living_measure.describe()
Out[22]:
Data distribution tells us, living_measure is right-skewed.
In [23]:
#Let's plot the boxplot for living_measure
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.boxplot(house_df.living_measure)
Out[23]:
There are many outliers in living measure. Need to review further to treat the same.
In [24]:
#checking the no. of data points with Living measure greater than 8000
house_df[house_df['living_measure']>8000]
Out[24]:
We have only 9 properties/house which have more than 8k living_measure. So will treat these outliers.
Analyzing Feature: lot_measure¶
In [25]:
#Data is skewed as visible from plot
plt.figure(figsize=(plotSizeX, plotSizeY))
print("Skewness is :",house_df.lot_measure.skew())
sns.boxplot(house_df.lot_measure)
house_df.lot_measure.describe()
Out[25]:
In [26]:
#checking the no. of data points with Lot measure greater than 1250000
house_df[house_df['lot_measure']>1250000]
Out[26]:
We have only 1 property with more than 12,50,000 lot_measure. So we need to treat this.
Analyzing Feature: ceil¶
In [27]:
#let's see the ceil count for all the records
house_df.ceil.value_counts()
Out[27]:
We can see, most houses have 1 floor
In [28]:
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.countplot('ceil',data=house_df)
Out[28]:
Above grapth confirming the same, that most properties have 1 and 2 floors
Analyzing Feature: coast¶
In [29]:
#coast - most houses donot have waterfront view, very few are waterfront
house_df.coast.value_counts()
Out[29]:
Analyzing Feature: sight¶
In [30]:
#sight - most sights have not been viewed
house_df.sight.value_counts()
Out[30]:
Analyzing Feature: condition¶
In [31]:
#condition - Overall most houses are rated as 3 and above for its condition overall
house_df.condition.value_counts()
Out[31]:
Analyzing Feature: quality¶
In [32]:
#Quality - most properties have quality rating between 6 to 10
house_df.quality.value_counts()
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.countplot('quality',data=house_df)
Out[32]:
In [33]:
#checking the no. of data points with quality rating as 13
house_df[house_df['quality']==13]
Out[33]:
There are only 13 propeties which have the highest quality rating
Analyzing Feature: ceil_measure¶
In [34]:
#ceil_measure - its highly skewed
print("Skewness is :", house_df.ceil_measure.skew())
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.distplot(house_df.ceil_measure)
house_df.ceil_measure.describe()
Out[34]:
In [35]:
sns.factorplot(x='ceil',y='ceil_measure',data=house_df, size = 4, aspect = 2)
Out[35]:
There is no pattern in Ceil Vs Ceil_measure
The vertival lines at each point represent the inter quartile range of values at that point
Analyzing Feature: basement¶
In [36]:
#basement_measure
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.distplot(house_df.basement)
Out[36]:
We can see 2 gaussians, which tells us there are propeties which don't have basements and some have the basements
In [37]:
house_df[house_df.basement==0].shape
Out[37]:
We have almost 60% of the properties without basement
In [38]:
#houses have zero measure of basement i.e. they donot have basements
#let's plot boxplot for properties which have basements only
house_df_base=house_df[house_df['basement']>0]
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.boxplot(house_df_base['basement'])
Out[38]:
We can clearly see, there are outliers. We need to treat this before our model.
In [39]:
#checking the no. of data points with 'basement' greater than 4000
house_df[house_df['basement']>4000]
Out[39]:
We have only 2 properties with more than 4,000 measure basement
In [40]:
#Distribution of houses having basement
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.distplot(house_df_base.basement)
Out[40]:
Distribution having basement is right-skewed
Analyzing Feature: yr_built¶
In [41]:
#house range from new to very old
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.distplot(house_df.yr_built)
Out[41]:
The built year of the properties range from 1900 to 2014 and we can see upward trend with time
Analyzing Feature: yr_renovated¶
In [42]:
house_df[house_df['yr_renovated']>0].shape
Out[42]:
Only 914 houses were renovated out of 21613 houses
In [43]:
#yr_renovated - plot of houses which are renovated
house_df_reno=house_df[house_df['yr_renovated']>0]
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.distplot(house_df_reno.yr_renovated)
Out[43]:
Now will create age column from columns : yr_built & yr_renovated
Analyzing Feature: Zipcode, Lat, Long¶
In [46]:
#For geographic visual
import geopandas as gpd
from shapely.geometry import Point, Polygon
#For current working directory
import os
cwd = os.getcwd()
In [47]:
## Need to add file USA ZipCodes_1.xlsx to current working directory to access this data
USAZip=pd.read_excel("USA ZipCodes_1.xlsx",sheet_name="Sheet8")
USAZip.head()
Out[47]:
In [48]:
house_df=house_df.merge(USAZip,how='left',on='zipcode')
#house_df.drop_duplicates()
In [49]:
#let's see the shape of our dataframe
house_df.shape
Out[49]:
Now we have 27 features
In [5]:
#Add the folder WA to your current working directory
usa = gpd.read_file(cwd+'\\WA\\WSDOT__City_Limits.shp')
usa.head()
gdf = gpd.GeoDataFrame(
house_df,geometry = [Point(xy) for xy in zip(house_df['long'], house_df['lat'])])
#We can now plot our ``GeoDataFrame``
ax=usa[usa.CityName.isin(house_df.City.unique())].plot(
color='white', edgecolor='black',figsize=(20,8))
plt.figure(figsize=(15,15))
gdf.plot(ax=ax, color='green', marker='o',markersize=0.1)
Out[5]:
In [51]:
#let's see the columns of dataframe once again
house_df.columns
Out[51]:
So we have 'City', 'Country', 'Type' as new feature in our dataframe
In [52]:
house_df.Type.value_counts()
Out[52]:
As the type is same for all the columns, we will remove this column in further analysis
In [53]:
house_df.City.value_counts()
Out[53]:
So we have most properties in 'Seattle' city and least in 'Medina' city
Analyzing Feature: furnished¶
In [54]:
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.countplot('furnished',data=house_df)
house_df.furnished.value_counts()
Out[54]:
Most properties are not furnished. Furnish column need to be converted into categorical column
BIVARIATE ANALYSIS¶
PairPlot¶
In [55]:
# let's plot all the variables and confirm our above deduction with more confidence
sns.pairplot(house_df, diag_kind = 'kde')
Out[55]:
From above pair plot, we observed/deduced below
- price: price distribution is Right-Skewed as we deduced earlier from our 5-factor analysis
- room_bed: our target variable (price) and room_bed plot is not linear. It's distribution have lot of gaussians
- room_bath: It's plot with price has somewhat linear relationship. Distribution has number of gaussians.
- living_measure: Plot against price has strong linear relationship. It also have linear relationship with room_bath variable. So might remove one of these 2. Distribution is Right-Skewed.
- lot_measure: No clear relationship with price.
- ceil: No clear relationship with price. We can see, it's have 6 unique values only. Therefore, we can convert this column into categorical column for values.
- coast: No clear relationship with price. Clearly it's categorical variable with 2 unique values.
- sight: No clear relationship with price. This has 5 unique values. Can be converted to Categorical variable.
- condition: No clear relationship with price. This has 5 unique values. Can be converted to Categorical variable.
- quality: Somewhat linear relationship with price. Has discrete values from 1 - 13. Can be converted to Categorical variable.
- ceil_measure: Strong linear relationship with price. Also with room_bath and living_measure features. Distribution is Right-Skewed.
- basement: No clear relationship with price.
- yr_built: No clear relationship with price.
- yr_renovated: No clear relationship with price. Have 2 unique values. Can be converted to Categorical Variable which tells whether house is renovated or not.
- zipcode, lat, long: No clear relationship with price or any other feature.
- living_measure15: Somewhat linear relationship with target feature. It's same as living_measure. Therefore we can drop this variable.
- lot_measure15: No clear relationship with price or any other feature.
- furnished: No clear relationship with price or any other feature. 2 unique values so can be converted to Categorical Variable
- total_area: No clear relationship with price. But it has Very Strong linear relationship with lot_measure. So one of it can be dropped.
In brief, below featues should be converted to Categorical Variable
ceil, coast, sight, condition, quality, yr_renovated, furnished
And below columns can be dropped after checking pearson factor
zipcode, lat, long, living_measure15, lot_measure15, total_areaIn [56]:
# let's see corelatoin between the different features
house_corr = house_df.corr(method ='pearson')
house_corr
Out[56]:
We have linear relationships in below featues as we got to know from above matrix
- price: room_bath, living_measure, quality, living_measure15, furnished
- living_measure: price, room_bath. So we can consider dropping 'room_bath' variable.
- quality: price, room_bath, living_measure
- ceil_measure: price, room_bath, living_measure, quality
- living_measure15: price, living_measure, quality. So we can consider dropping living_measure15 as well. As it's giving same info as living_measure.
- lot_measure15: lot_measure. Therefore, we can consider dropping lot_measure15, as it's giving same info.
- furnished: quality
- total_area: lot_measure, lot_measure15. Therefore, we can consider dropping total_area feature as well. As it's giving same info as lot_measure.
We can plot heatmap and can easily confirm our above findings
In [57]:
# Plotting heatmap
plt.subplots(figsize =(15, 8))
sns.heatmap(house_corr,cmap="YlGnBu",annot=True)
Out[57]:
Analyzing Bivariate for Feature: month_year¶
In [58]:
#month,year in which house is sold. Price is not influenced by it, though there are outliers and can be easily seen.
house_df['month_year'] = pd.to_datetime(house_df['month_year'], format='%B-%Y')
house_df.sort_values(["month_year"], axis=0,
ascending=True, inplace=True)
house_df["month_year"] = house_df["month_year"].dt.strftime('%B-%Y')
sns.factorplot(x='month_year',y='price',data=house_df, size=4, aspect=2)
plt.xticks(rotation=90)
#groupby
house_df.groupby('month_year')['price'].agg(['mean','median','size'])
Out[58]:
The mean price of the houses tend to be high during March,April, May as compared to that of September, October, November,December period.
Analyzing Bivariate for Feature: room_bed¶
In [59]:
#Room_bed - outliers can be seen easily. Mean and median of price increases with number bedrooms/house uptill a point
#and then drops
sns.factorplot(x='room_bed',y='price',data=house_df, size=4, aspect=2)
#groupby
house_df.groupby('room_bed')['price'].agg(['mean','median','size'])
Out[59]:
There is clear increasing trend in price with room_bed
In [60]:
#room_bath - outliers can be seen easily. Overall mean and median price increares with increasing room_bath
sns.factorplot(x='room_bath',y='price',data=house_df,size=4, aspect=2)
plt.xticks(rotation=90)
#groupby
house_df.groupby('room_bath')['price'].agg(['mean','median','size'])
Out[60]:
There is upward trend in price with increase in room_bath
Analyzing Bivariate for Feature: living_measure¶
In [61]:
#living_measure - price increases with increase in living measure
plt.figure(figsize=(plotSizeX, plotSizeY))
print(sns.scatterplot(house_df['living_measure'],house_df['price']))
house_df['living_measure'].describe()
Out[61]:
There is clear increment in price of the property with increment in the living measure But there seems to be one outlier to this trend. Need to evaluate the same
Analyzing Bivariate for Feature: lot_measure¶
In [62]:
#lot_measure - there seems to be no relation between lot_measure and price
#lot_measure - data value range is very large so breaking it get better view.
plt.figure(figsize=(plotSizeX, plotSizeY))
print(sns.scatterplot(house_df['lot_measure'],house_df['price']))
house_df['lot_measure'].describe()
Out[62]:
There doesnt seem to be no relation between lot_measure and price trend
In [63]:
#lot_measure <25000
plt.figure(figsize=(plotSizeX, plotSizeY))
x=house_df[house_df['lot_measure']<25000]
print(sns.scatterplot(x['lot_measure'],x['price']))
x['lot_measure'].describe()
Out[63]:
Almost 95% of the houses have <25000 lot_measure. But there is no clear trend between lot_measure and price
In [64]:
#lot_measure >100000 - price increases with increase in living measure
plt.figure(figsize=(plotSizeX, plotSizeY))
y=house_df[house_df['lot_measure']<=75000]
print(sns.scatterplot(y['lot_measure'],y['price']))
#y['lot_measure'].describe()
Analyzing Bivariate for Feature: ceil¶
In [65]:
#ceil - median price increases initially and then falls
print(sns.factorplot(x='ceil',y='price',data=house_df, size = 4, aspect = 2))
#groupby
house_df.groupby('ceil')['price'].agg(['mean','median','size'])
Out[65]:
There is some slight upward trend in price with the ceil
Analyzing Bivariate for Feature: coast¶
In [66]:
#coast - mean and median of waterfront view is high however such houses are very small in compare to non-waterfront
#Also, living_measure mean and median is greater for waterfront house.
print(sns.factorplot(x='coast',y='price',data=house_df, size = 4, aspect = 2))
#groupby
house_df.groupby('coast')['living_measure','price'].agg(['median','mean'])
Out[66]:
The house properties with water_front tend to have higher price compared to that of non-water_front properties
Analyzing Bivariate for Feature: sight¶
In [67]:
#sight - have outliers. The house sighted more have high price (mean and median) and have large living area as well.
print(sns.factorplot(x='sight',y='price',data=house_df, size = 4, aspect = 2))
#groupby
house_df.groupby('sight')['price','living_measure'].agg(['mean','median','size'])
Out[67]:
Properties with higher price have more no.of sights compared to that of houses with lower price
In [68]:
#Sight - Viewed in relation with price and living_measure
#Costlier houses with large living area are sighted more.
plt.figure(figsize=(plotSizeX, plotSizeY))
print(sns.scatterplot(house_df['living_measure'],house_df['price'],hue=house_df['sight'],palette='Paired',legend='full'))
The above graph also justify that: Properties with higher price have more no.of sights compared to that of houses with lower price
Analyzing Bivariate for Feature: condition¶
In [69]:
#condition - as the condition rating increases its price and living measure mean and median also increases.
print(sns.factorplot(x='condition',y='price',data=house_df, size = 4, aspect = 2))
#groupby
house_df.groupby('condition')['price','living_measure'].agg(['mean','median','size'])
Out[69]:
The price of the house increases with condition rating of the house
In [70]:
#Condition - Viewed in relation with price and living_measure. Most houses are rated as 3 or more.
#We can see some outliers as well
plt.figure(figsize=(plotSizeX, plotSizeY))
print(sns.scatterplot(house_df['living_measure'],house_df['price'],hue=house_df['condition'],palette='Paired',legend='full'))
So we found out that smaller houses are in better condition and better condition houses are having higher prices
Analyzing Bivariate for Feature: quality¶
In [71]:
#quality - with grade increase price and living_measure increase (mean and median)
print(sns.factorplot(x='quality',y='price',data=house_df, size = 4, aspect = 2))
#groupby
house_df.groupby('quality')['price','living_measure'].agg(['mean','median','size'])
Out[71]:
There is clear increase in price of the house with higher rating on quality
In [72]:
#quality - Viewed in relation with price and living_measure. Most houses are graded as 6 or more.
#We can see some outliers as well
plt.figure(figsize=(plotSizeX, plotSizeY))
print(sns.scatterplot(house_df['living_measure'],house_df['price'],hue=house_df['quality'],palette='coolwarm_r',
legend='full'))
Analyzing Bivariate for Feature: ceil_measure¶
In [73]:
#ceil_measure - price increases with increase in ceil measure
plt.figure(figsize=(plotSizeX, plotSizeY))
print(sns.scatterplot(house_df['ceil_measure'],house_df['price']))
house_df['ceil_measure'].describe()
Out[73]:
There is upward trend in price with ceil_measure
Analyzing Bivariate for Feature: basement¶
In [74]:
#basement - price increases with increase in ceil measure
plt.figure(figsize=(plotSizeX, plotSizeY))
print(sns.scatterplot(house_df['basement'],house_df['price']))
house_df['basement'].describe()
Out[74]:
We will create the categorical variable for basement 'has_basement' for houses with basement and no basement.This categorical variable will be used for further analysis.
In [75]:
#Binning Basement to analyse data
def create_basement_group(series):
if series == 0:
return "No"
elif series > 0:
return "Yes"
house_df['has_basement'] = house_df['basement'].apply(create_basement_group)
In [76]:
#basement - after binning we data shows with basement houses are costlier and have higher
#living measure (mean & median)
print(sns.factorplot(x='has_basement',y='price',data=house_df, size = 4, aspect = 2))
house_df.groupby('has_basement')['price','living_measure'].agg(['mean','median','size'])
Out[76]:
The houses with basement has better price compared to that of houses without basement
In [77]:
#basement - have higher price & living measure
plt.figure(figsize=(plotSizeX, plotSizeY))
print(sns.scatterplot(house_df['living_measure'],house_df['price'],hue=house_df['has_basement']))
In [78]:
#yr_built - outliers can be seen easily.
plt.figure(figsize=(plotSizeX, plotSizeY))
print(sns.scatterplot(house_df['yr_built'],house_df['living_measure']))
#groupby
house_df.groupby('yr_built')['price'].agg(['mean','median','size'])
Out[78]:
We will create new variable: Houselandratio - This is proportion of living area in the total area of the house. We will explore the trend of price against this houselandratio.
In [79]:
#HouseLandRatio - Computing new variable as ratio of living_measure/total_area
#Significes - Land used for construction of house
house_df["HouseLandRatio"]=np.round((house_df['living_measure']/house_df['total_area']),2)*100
house_df["HouseLandRatio"].head()
Out[79]:
Analyzing Bivariate for Feature: yr_renovated¶
In [80]:
#yr_renovated -
plt.figure(figsize=(plotSizeX, plotSizeY))
x=house_df[house_df['yr_renovated']>0]
print(sns.scatterplot(x['yr_renovated'],x['price']))
#groupby
x.groupby('yr_renovated')['price'].agg(['mean','median','size'])
Out[80]:
So most houses are renovated after 1980's. We will create new categorical variable 'has_renovated' to categorize the property as renovated and non-renovated. For further ananlysis we will use this categorical variable.
In [81]:
#Lets try to group yr_renovated
#Binning Basement to analyse data
def create_renovated_group(series):
if series == 0:
return "No"
elif series > 0:
return "Yes"
house_df['has_renovated'] = house_df['yr_renovated'].apply(create_renovated_group)
In [84]:
#has_renovated - renovated have higher mean and median, however it does not confirm if the prices of house renovated
#actually increased or not.
#HouseLandRatio - Renovated house utilized more land area for construction of house
plt.figure(figsize=(plotSizeX, plotSizeY))
print(sns.scatterplot(house_df['living_measure'],house_df['price'],hue=house_df['has_renovated']))
#groupby
house_df.groupby(['has_renovated'])['price','HouseLandRatio'].agg(['mean','median','size'])
Out[84]:
Renovated properties have higher price than others with same living measure space.
In [85]:
#pd.crosstab(house_df['yearbuilt_group'],house_df['has_renovated'])
In [86]:
#has_renovated - have higher price & living measure
plt.figure(figsize=(plotSizeX, plotSizeY))
x=house_df[house_df['yr_built']<2000]
print(sns.scatterplot(x['living_measure'],x['price'],hue=x['has_renovated']))
Analyzing Bivariate for Feature: furnished¶
In [87]:
#furnished - Furnished has higher price value and has greater living_measure
plt.figure(figsize=(plotSizeX, plotSizeY))
print(sns.scatterplot(house_df['living_measure'],house_df['price'],hue=house_df['furnished']))
#groupby
house_df.groupby('furnished')['price','living_measure','HouseLandRatio'].agg(['mean','median','size'])
Out[87]:
Furnished houses have higher price than that of the Non-furnished houses
Analyzing Bivariate for Feature: city¶
In [88]:
#City - outliers can be seen easily.
print(sns.factorplot(x='City',y='price',data=house_df, size = 4, aspect = 2))
plt.xticks(rotation=90)
#groupby
house_df.groupby('City')['price'].agg(['mean','median','size']).sort_values(by='median',ascending=False)
Out[88]:
From the above graph, few cities have higher average price of the houses compared to others. We need to further analyse why the price varies among cities.
In [89]:
#City mean price distribution with average
city_price=pd.DataFrame(house_df.groupby('City')['price'].agg(['mean','median','size']))
indx=city_price.index
overall_price_mean=np.mean(house_df['price'])
overall_price_median=np.median(house_df['price'])
fig, ax1 = plt.subplots(figsize=(plotSizeX, plotSizeY))
barlist=ax1.bar(city_price.index,city_price['mean'],color='gray')
plt.xticks(rotation=90)
ax1.axhline(overall_price_mean, color="red")
ax1.text(1.02, overall_price_mean, "{0:.2f}".format(round(overall_price_mean,2)), va='center', ha="left", bbox=dict(facecolor="w",alpha=0.5),
transform=ax1.get_yaxis_transform())
plt.title("Cities and Mean Price")
plt.show()
As we can see from above grapgh, majorly below cities have higher mean house prices
- Bellevue
- Fall City
- Federal Way
- Kirkland
- Medina
- Mercer Island
- Redmond
- Sammanmish
- Woodinville
In [90]:
#City median price distribution with average
fig, ax1 = plt.subplots(figsize=(plotSizeX, plotSizeY))
barlist=ax1.bar(city_price.index,city_price['median'],color='green')
plt.xticks(rotation=90)
ax1.axhline(overall_price_median, color="red")
ax1.text(1.02, overall_price_median, "{0:.2f}".format(round(overall_price_median,2)), va='center', ha="left", bbox=dict(facecolor="w",alpha=0.5),
transform=ax1.get_yaxis_transform())
plt.title("Cities and Median Price")
plt.show()
As we can see from above grapgh, majorly below cities have higher median house prices
- Bellevue
- Bothell
- Issaquah
- Kirkland
- Medina
- Mercer Island
- Redmond
- Sammanmish
- Snoqualmie
- Woodinville
In [91]:
#let's make the copy of the dataframe, before making any furhter changes
house_df_bdp=house_df.copy()
DATA PROCESSING¶
Treating Outlilers¶
We have seen outliers for columns room_bath(33 bed), living_measure, lot_measure, ceil_measure and Basement
In [92]:
def outlier_treatment(datacolumn):
sorted(datacolumn)
Q1,Q3 = np.percentile(datacolumn , [25,75])
IQR = Q3-Q1
lower_range = Q1-(1.5 * IQR)
upper_range = Q3+(1.5 * IQR)
return lower_range,upper_range
Using the above function, lets get the lowerbound and upperbound values
Treating outliers for column - ceil_measure¶
In [93]:
lowerbound,upperbound = outlier_treatment(house_df.ceil_measure)
print(lowerbound,upperbound)
Lets check which column is considered as an outlier
In [94]:
house_df[(house_df.ceil_measure < lowerbound) | (house_df.ceil_measure > upperbound)]
Out[94]:
We got 611 records which are outliers
In [95]:
#dropping the record from the dataset
house_df.drop(house_df[ (house_df.ceil_measure > upperbound) | (house_df.ceil_measure < lowerbound) ].index, inplace=True)
In [96]:
house_df.shape
Out[96]:
In [97]:
#ceil_measure
print("Skewness is :", house_df.ceil_measure.skew())
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.distplot(house_df.ceil_measure)
house_df.ceil_measure.describe()
Out[97]:
After treating outliers of ceil_measure, the data has reduced by about 600(~3%) data points but data is nicely distributed
Treating outliers for column - basement¶
In [98]:
lowerbound_base,upperbound_base = outlier_treatment(house_df.basement)
print(lowerbound_base,upperbound_base)
In [99]:
house_df[(house_df.basement < lowerbound_base) | (house_df.basement > upperbound_base)]
Out[99]:
We got 408 records as outliers, let's drop these outliers
In [100]:
#dropping the record from the dataset
house_df.drop(house_df[ (house_df.basement > upperbound_base) | (house_df.basement < lowerbound_base) ].index, inplace=True)
In [101]:
house_df.shape
Out[101]:
In [102]:
#basement_measure
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.distplot(house_df.basement)
Out[102]:
After treating outliers of basement, we can see that 400(~2%) data points got imputed. Total about 5% data has been imputed after treating ceil_measure and basement.
In [103]:
#Let's see the boxplot now for basement
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.boxplot(house_df['basement'])
Out[103]:
Treating outliers for column - living_measure¶
In [104]:
lowerbound_lim,upperbound_lim = outlier_treatment(house_df.living_measure)
print(lowerbound_lim,upperbound_lim)
In [105]:
house_df[(house_df.living_measure < lowerbound_lim) | (house_df.living_measure > upperbound_lim)]
Out[105]:
We got 178 records as outliers. Let's treat this by dropping
In [106]:
#dropping the record from the dataset
house_df.drop(house_df[ (house_df.living_measure > upperbound_lim) | (house_df.living_measure < lowerbound_lim) ].index, inplace=True)
In [107]:
#let's see the boxplot after dropping the outliers
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.boxplot(house_df['living_measure'])
Out[107]:
In [108]:
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.distplot(house_df.living_measure)
Out[108]:
By treating outliers of living_measure, we lost 178 data points more and data distribution looks normal
In [109]:
# shape of the data after imputing outliers in living_column
house_df.shape
Out[109]:
Treating outliers for column - lot_measure¶
In [110]:
lowerbound_lom,upperbound_lom = outlier_treatment(house_df.lot_measure)
print(lowerbound_lom,upperbound_lom)
In [111]:
house_df[(house_df.lot_measure < lowerbound_lom) | (house_df.lot_measure > upperbound_lom)]
Out[111]:
We got 2155 records which are outliers. Let's drop these outlier records.
In [112]:
#dropping the record from the dataset
house_df.drop(house_df[ (house_df.lot_measure > upperbound_lom) | (house_df.lot_measure < lowerbound_lom) ].index, inplace=True)
In [113]:
#let's plot after treating outliers
plt.figure(figsize=(plotSizeX, plotSizeY))
sns.boxplot(house_df['lot_measure'])
Out[113]:
In [114]:
house_df.shape
Out[114]:
Total outliers in the lot_measure are 2128 data points. But still we are going ahead with imputing the data. We will analyze later whether there is any impact on the data set or not.
Treating outliers for column - room_bed¶
In [115]:
#As we know for room_bed = 33 was outlier from our earlier findings, let's see the record and drop it
house_df[house_df['room_bed']==33]
Out[115]:
In [116]:
#dropping the record from the dataset
house_df.drop(house_df[ (house_df.room_bed == 33) ].index, inplace=True)
In [117]:
house_df.shape
Out[117]:
In summary, after treating outliers, we have lost about 15% of the data. We will analyse the impact of this data loss during the model evaluation.¶
In [118]:
#let's see the feature/columns and drop the unneccessary features
house_df.columns
Out[118]:
As we already have this information in other features. We will drop the unwanted columns from new copied dataframe instance : cid,dayhours,yr_renovated,zipcode,lat,long,county,type
In [119]:
#Let's create another dataframe for modeling
df_model=house_df.copy()
In [120]:
#let's check the new copy of dataframe by printing first few records
df_model.head()
Out[120]:
New instance of dataframe for model created successfully
In [121]:
#let's verify the columns
df_model.columns
Out[121]:
In [122]:
#Dropping the feature not required in 1st Iteration
df_final=df_model.drop(['cid','dayhours','yr_renovated','zipcode','lat','long','County','Type'],axis=1)
In [123]:
df_final.shape
Out[123]:
In [124]:
df_final.head()
Out[124]:
In [125]:
df_final.columns
Out[125]:
Creating dummies for categorical variables: 'room_bed', 'room_bath', 'ceil', 'coast', 'sight', 'condition', 'quality', 'furnished','City', 'has_basement', 'has_renovated'¶
In [126]:
# Getting dummies for columns ceil, coast, sight, condition, quality, yr_renovated, furnished
dff = pd.get_dummies(df_final, columns=['room_bed', 'room_bath', 'ceil', 'coast', 'sight', 'condition', 'quality', 'furnished','City',
'has_basement', 'has_renovated'],drop_first=True)
In [127]:
# let's see the data types of the features
dff.shape
Out[127]:
In [128]:
dff.columns
Out[128]:
Ready for model building¶
'dff' is the data frame which is ready for modeling¶
In [129]:
dff.head()
Out[129]:
In [130]:
#let's drop the month_year column as we already analyzed it
dff=dff.drop(['month_year'],axis=1)
In [131]:
#Creating X, y for training and testing set
X = dff.drop("price" , axis=1)
y = dff["price"]
In [132]:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=10)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=10)
In [133]:
print(X_train.shape)
print(X_test.shape)
print(X_val.shape)
In [134]:
dff.head()
Out[134]:
Model building¶
Let's build the model and see their performances
Linear Regression (with Ridge and Lasso)¶
In [135]:
#importing the necessary libraries
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import Ridge
from sklearn.linear_model import Lasso
from sklearn import metrics
from sklearn.metrics import r2_score, mean_squared_error, mean_absolute_error
In [136]:
LR1 = LinearRegression()
LR1.fit(X_train, y_train)
#predicting result over test data
y_LR1_predtr= LR1.predict(X_train)
y_LR1_predvl= LR1.predict(X_val)
LR1.coef_
Out[136]:
In [137]:
#Model score and Deduction for each Model in a DataFrame
LR1_trscore=r2_score(y_train,y_LR1_predtr)
LR1_trRMSE=np.sqrt(mean_squared_error(y_train, y_LR1_predtr))
LR1_trMSE=mean_squared_error(y_train, y_LR1_predtr)
LR1_trMAE=mean_absolute_error(y_train, y_LR1_predtr)
LR1_vlscore=r2_score(y_val,y_LR1_predvl)
LR1_vlRMSE=np.sqrt(mean_squared_error(y_val, y_LR1_predvl))
LR1_vlMSE=mean_squared_error(y_val, y_LR1_predvl)
LR1_vlMAE=mean_absolute_error(y_val, y_LR1_predvl)
Compa_df=pd.DataFrame({'Method':['Linear Reg Model1'],'Val Score':LR1_vlscore,'RMSE_vl': LR1_vlRMSE, 'MSE_vl': LR1_vlMSE, 'MAE_vl': LR1_vlMAE,'train Score':LR1_trscore,'RMSE_tr': LR1_trRMSE, 'MSE_tr': LR1_trMSE, 'MAE_tr': LR1_trMAE})
#Compa_df = Compa_df[['Method', 'Test Score', 'RMSE', 'MSE', 'MAE']]
Compa_df
Out[137]:
The linear regression model performed with scores 0.73 & .72 in training data set and validation data set respectively
In [138]:
sns.set(style="darkgrid", color_codes=True)
with sns.axes_style("white"):
sns.jointplot(x=y_val, y=y_LR1_predvl, kind="reg", color="k")
Lasso model¶
In [139]:
Lasso1 = Lasso(alpha=1)
Lasso1.fit(X_train, y_train)
#predicting result over test data
y_Lasso1_predtr= Lasso1.predict(X_train)
y_Lasso1_predvl= Lasso1.predict(X_val)
Lasso1.coef_
Out[139]:
In [140]:
#Model score and Deduction for each Model in a DataFrame
Lasso1_trscore=r2_score(y_train,y_Lasso1_predtr)
Lasso1_trRMSE=np.sqrt(mean_squared_error(y_train, y_Lasso1_predtr))
Lasso1_trMSE=mean_squared_error(y_train, y_Lasso1_predtr)
Lasso1_trMAE=mean_absolute_error(y_train, y_Lasso1_predtr)
Lasso1_vlscore=r2_score(y_val,y_Lasso1_predvl)
Lasso1_vlRMSE=np.sqrt(mean_squared_error(y_val, y_Lasso1_predvl))
Lasso1_vlMSE=mean_squared_error(y_val, y_Lasso1_predvl)
Lasso1_vlMAE=mean_absolute_error(y_val, y_Lasso1_predvl)
Lasso1_df=pd.DataFrame({'Method':['Linear-Reg Lasso1'],'Val Score':Lasso1_vlscore,'RMSE_vl': Lasso1_vlRMSE, 'MSE_vl': Lasso1_vlMSE, 'MAE_vl': Lasso1_vlMAE,'train Score':Lasso1_trscore,'RMSE_tr': Lasso1_trRMSE, 'MSE_tr': Lasso1_trMSE, 'MAE_tr': Lasso1_trMAE})
Compa_df = pd.concat([Compa_df, Lasso1_df])
Compa_df
Out[140]:
The lasso linear regression model performed with scores 0.73 & .72 in training data set and validation data set respectively. The coefficeints of 1 variable in lasso model is almost '0', signifying that the variable with '0' coefficient can be dropped.
In [141]:
sns.set(style="darkgrid", color_codes=True)
with sns.axes_style("white"):
sns.jointplot(x=y_val, y=y_Lasso1_predvl, kind="reg", color="k")
Ridge model¶
In [142]:
Ridge1 = Ridge(alpha=0.5)
Ridge1.fit(X_train, y_train)
#predicting result over test data
y_Ridge1_predtr= Ridge1.predict(X_train)
y_Ridge1_predvl= Ridge1.predict(X_val)
Ridge1.coef_
Out[142]:
In [143]:
#Model score and Deduction for each Model in a DataFrame
Ridge1_trscore=r2_score(y_train,y_Ridge1_predtr)
Ridge1_trRMSE=np.sqrt(mean_squared_error(y_train, y_Ridge1_predtr))
Ridge1_trMSE=mean_squared_error(y_train, y_Ridge1_predtr)
Ridge1_trMAE=mean_absolute_error(y_train, y_Ridge1_predtr)
Ridge1_vlscore=r2_score(y_val,y_Ridge1_predvl)
Ridge1_vlRMSE=np.sqrt(mean_squared_error(y_val, y_Ridge1_predvl))
Ridge1_vlMSE=mean_squared_error(y_val, y_Ridge1_predvl)
Ridge1_vlMAE=mean_absolute_error(y_val, y_Ridge1_predvl)
Ridge1_df=pd.DataFrame({'Method':['Linear-Reg Ridge1'],'Val Score':Ridge1_vlscore,'RMSE_vl': Ridge1_vlRMSE, 'MSE_vl': Ridge1_vlMSE, 'MAE_vl': Ridge1_vlMAE,'train Score':Ridge1_trscore,'RMSE_tr': Ridge1_trRMSE, 'MSE_tr': Ridge1_trMSE, 'MAE_tr': Ridge1_trMAE})
Compa_df = pd.concat([Compa_df, Ridge1_df])
Compa_df
Out[143]:
The Ridge linear regression model performed with scores 0.73 & .72 in training data set and validation data set respectively. The coefficeints of variables in ridge model are all non-zero, indicating that non of the variables can be dropped.
In [144]:
sns.set(style="darkgrid", color_codes=True)
with sns.axes_style("white"):
sns.jointplot(x=y_val, y=y_Ridge1_predvl, kind="reg", color="k")
In summary, Linear models have performed almost with similar results in both regularized model and non-regularized models
KNN Regressor¶
In [145]:
from sklearn.neighbors import KNeighborsRegressor
In [146]:
knn1 = KNeighborsRegressor(n_neighbors=4,weights='distance')
knn1.fit(X_train, y_train)
#predicting result over test data
y_knn1_predtr= knn1.predict(X_train)
y_knn1_predvl= knn1.predict(X_val)
In [147]:
#Model score and Deduction for each Model in a DataFrame
knn1_trscore=r2_score(y_train,y_knn1_predtr)
knn1_trRMSE=np.sqrt(mean_squared_error(y_train, y_knn1_predtr))
knn1_trMSE=mean_squared_error(y_train, y_knn1_predtr)
knn1_trMAE=mean_absolute_error(y_train, y_knn1_predtr)
knn1_vlscore=r2_score(y_val,y_knn1_predvl)
knn1_vlRMSE=np.sqrt(mean_squared_error(y_val, y_knn1_predvl))
knn1_vlMSE=mean_squared_error(y_val, y_knn1_predvl)
knn1_vlMAE=mean_absolute_error(y_val, y_knn1_predvl)
knn1_df=pd.DataFrame({'Method':['knn1'],'Val Score':knn1_vlscore,'RMSE_vl': knn1_vlRMSE, 'MSE_vl': knn1_vlMSE, 'MAE_vl': knn1_vlMAE,'train Score':knn1_trscore,'RMSE_tr': knn1_trRMSE, 'MSE_tr': knn1_trMSE, 'MAE_tr': knn1_trMAE})
Compa_df = pd.concat([Compa_df, knn1_df])
Compa_df
Out[147]:
Though KNN regressor performed well in training set, the performance score in validation set is very less. This shows that the model is overfitted in training set
Support vector regressor¶
In [148]:
from sklearn.svm import SVR
In [149]:
SVR1 = SVR(gamma='auto',C=10.0, epsilon=0.2,kernel='rbf')
SVR1.fit(X_train, y_train)
y_SVR1_predtr= SVR1.predict(X_train)
y_SVR1_predvl= SVR1.predict(X_val)
In [150]:
#Model score and Deduction for each Model in a DataFrame
SVR1_trscore=r2_score(y_train,y_SVR1_predtr)
SVR1_trRMSE=np.sqrt(mean_squared_error(y_train, y_SVR1_predtr))
SVR1_trMSE=mean_squared_error(y_train, y_SVR1_predtr)
SVR1_trMAE=mean_absolute_error(y_train, y_SVR1_predtr)
SVR1_vlscore=r2_score(y_val,y_SVR1_predvl)
SVR1_vlRMSE=np.sqrt(mean_squared_error(y_val, y_SVR1_predvl))
SVR1_vlMSE=mean_squared_error(y_val, y_SVR1_predvl)
SVR1_vlMAE=mean_absolute_error(y_val, y_SVR1_predvl)
SVR1_df=pd.DataFrame({'Method':['SVR1'],'Val Score':SVR1_vlscore,'RMSE_vl': SVR1_vlRMSE, 'MSE_vl': SVR1_vlMSE, 'MAE_vl': SVR1_vlMAE,'train Score':SVR1_trscore,'RMSE_tr': SVR1_trRMSE, 'MSE_tr': SVR1_trMSE, 'MAE_tr': SVR1_trMAE})
Compa_df = pd.concat([Compa_df, SVR1_df])
Compa_df
Out[150]:
The above negative scores in SVR model is due to non-learning of the model in the training set which results in non-performance in validation set
In [151]:
SVR2 = SVR(gamma='auto',C=0.1,kernel='linear')
SVR2.fit(X_train, y_train)
y_SVR2_predtr= SVR2.predict(X_train)
y_SVR2_predvl= SVR2.predict(X_val)
#Model score and Deduction for each Model in a DataFrame
SVR2_trscore=r2_score(y_train,y_SVR2_predtr)
SVR2_trRMSE=np.sqrt(mean_squared_error(y_train, y_SVR2_predtr))
SVR2_trMSE=mean_squared_error(y_train, y_SVR2_predtr)
SVR2_trMAE=mean_absolute_error(y_train, y_SVR2_predtr)
SVR2_vlscore=r2_score(y_val,y_SVR2_predvl)
SVR2_vlRMSE=np.sqrt(mean_squared_error(y_val, y_SVR2_predvl))
SVR2_vlMSE=mean_squared_error(y_val, y_SVR2_predvl)
SVR2_vlMAE=mean_absolute_error(y_val, y_SVR2_predvl)
SVR2_df=pd.DataFrame({'Method':['SVR2'],'Val Score':SVR2_vlscore,'RMSE_vl': SVR2_vlRMSE, 'MSE_vl': SVR2_vlMSE, 'MAE_vl': SVR2_vlMAE,'train Score':SVR2_trscore,'RMSE_tr': SVR2_trRMSE, 'MSE_tr': SVR2_trMSE, 'MAE_tr': SVR2_trMAE})
Compa_df = pd.concat([Compa_df, SVR2_df])
Compa_df
Out[151]:
The SVR model with modified parameters has not performed well with just ~0.45 in both training and validation data sets
Decision Tree Regressor¶
In [152]:
from sklearn.tree import DecisionTreeRegressor
In [153]:
DT1 = DecisionTreeRegressor()
DT1.fit(X_train, y_train)
y_DT1_predtr= DT1.predict(X_train)
y_DT1_predvl= DT1.predict(X_val)
#Model score and Deduction for each Model in a DataFrame
DT1_trscore=r2_score(y_train,y_DT1_predtr)
DT1_trRMSE=np.sqrt(mean_squared_error(y_train, y_DT1_predtr))
DT1_trMSE=mean_squared_error(y_train, y_DT1_predtr)
DT1_trMAE=mean_absolute_error(y_train, y_DT1_predtr)
DT1_vlscore=r2_score(y_val,y_DT1_predvl)
DT1_vlRMSE=np.sqrt(mean_squared_error(y_val, y_DT1_predvl))
DT1_vlMSE=mean_squared_error(y_val, y_DT1_predvl)
DT1_vlMAE=mean_absolute_error(y_val, y_DT1_predvl)
DT1_df=pd.DataFrame({'Method':['DT1'],'Val Score':DT1_vlscore,'RMSE_vl': DT1_vlRMSE, 'MSE_vl': DT1_vlMSE, 'MAE_vl': DT1_vlMAE,'train Score':DT1_trscore,'RMSE_tr': DT1_trRMSE, 'MSE_tr': DT1_trMSE, 'MAE_tr': DT1_trMAE})
Compa_df = pd.concat([Compa_df, DT1_df])
Compa_df
Out[153]:
Above performance of initial Decision tree model shows overfit in training set with 0.99 score and low performance in validation set
In [154]:
DT2 = DecisionTreeRegressor(max_depth=10,min_samples_leaf=5)
DT2.fit(X_train, y_train)
y_DT2_predtr= DT2.predict(X_train)
y_DT2_predvl= DT2.predict(X_val)
#Model score and Deduction for each Model in a DataFrame
DT2_trscore=r2_score(y_train,y_DT2_predtr)
DT2_trRMSE=np.sqrt(mean_squared_error(y_train, y_DT2_predtr))
DT2_trMSE=mean_squared_error(y_train, y_DT2_predtr)
DT2_trMAE=mean_absolute_error(y_train, y_DT2_predtr)
DT2_vlscore=r2_score(y_val,y_DT2_predvl)
DT2_vlRMSE=np.sqrt(mean_squared_error(y_val, y_DT2_predvl))
DT2_vlMSE=mean_squared_error(y_val, y_DT2_predvl)
DT2_vlMAE=mean_absolute_error(y_val, y_DT2_predvl)
DT2_df=pd.DataFrame({'Method':['DT2'],'Val Score':DT2_vlscore,'RMSE_vl': DT2_vlRMSE, 'MSE_vl': DT2_vlMSE, 'MAE_vl': DT2_vlMAE,'train Score':DT2_trscore,'RMSE_tr': DT2_trRMSE, 'MSE_tr': DT2_trMSE, 'MAE_tr': DT2_trMAE})
Compa_df = pd.concat([Compa_df, DT2_df])
Compa_df
Out[154]:
Above decision tree model with modified parameter has better performed on the training set and validation set compared to initial decision tree model.But overall decision tree has not performed well than linear regression models.
In [155]:
sns.set(style="darkgrid", color_codes=True)
with sns.axes_style("white"):
sns.jointplot(x=y_val, y=y_DT2_predvl, kind="reg", color="k")
In summary, KNN regressor model and decision tree models have not performed well in comparison with linear regression models
Ensemble techniques¶
Boosting and Bagging¶
In [156]:
from sklearn.ensemble import GradientBoostingRegressor, BaggingRegressor
In [157]:
GB1=GradientBoostingRegressor(n_estimators = 200, learning_rate = 0.1, random_state=22)
GB1.fit(X_train, y_train)
y_GB1_predtr= GB1.predict(X_train)
y_GB1_predvl= GB1.predict(X_val)
#Model score and Deduction for each Model in a DataFrame
GB1_trscore=r2_score(y_train,y_GB1_predtr)
GB1_trRMSE=np.sqrt(mean_squared_error(y_train, y_GB1_predtr))
GB1_trMSE=mean_squared_error(y_train, y_GB1_predtr)
GB1_trMAE=mean_absolute_error(y_train, y_GB1_predtr)
GB1_vlscore=r2_score(y_val,y_GB1_predvl)
GB1_vlRMSE=np.sqrt(mean_squared_error(y_val, y_GB1_predvl))
GB1_vlMSE=mean_squared_error(y_val, y_GB1_predvl)
GB1_vlMAE=mean_absolute_error(y_val, y_GB1_predvl)
GB1_df=pd.DataFrame({'Method':['GB1'],'Val Score':GB1_vlscore,'RMSE_vl': GB1_vlRMSE, 'MSE_vl': GB1_vlMSE, 'MAE_vl': GB1_vlMAE,'train Score':GB1_trscore,'RMSE_tr': GB1_trRMSE, 'MSE_tr': GB1_trMSE, 'MAE_tr': GB1_trMAE})
Compa_df = pd.concat([Compa_df, GB1_df])
Compa_df
Out[157]:
Gradient boosting model has provided good scores in both training and validation sets
In [158]:
BGG1=BaggingRegressor(n_estimators=50, oob_score= True,random_state=14)
BGG1.fit(X_train, y_train)
y_BGG1_predtr= BGG1.predict(X_train)
y_BGG1_predvl= BGG1.predict(X_val)
#Model score and Deduction for each Model in a DataFrame
BGG1_trscore=r2_score(y_train,y_BGG1_predtr)
BGG1_trRMSE=np.sqrt(mean_squared_error(y_train, y_BGG1_predtr))
BGG1_trMSE=mean_squared_error(y_train, y_BGG1_predtr)
BGG1_trMAE=mean_absolute_error(y_train, y_BGG1_predtr)
BGG1_vlscore=r2_score(y_val,y_BGG1_predvl)
BGG1_vlRMSE=np.sqrt(mean_squared_error(y_val, y_BGG1_predvl))
BGG1_vlMSE=mean_squared_error(y_val, y_BGG1_predvl)
BGG1_vlMAE=mean_absolute_error(y_val, y_BGG1_predvl)
BGG1_df=pd.DataFrame({'Method':['BGG1'],'Val Score':BGG1_vlscore,'RMSE_vl': BGG1_vlRMSE, 'MSE_vl':BGG1_vlMSE, 'MAE_vl': BGG1_vlMAE,'train Score':BGG1_trscore,'RMSE_tr': BGG1_trRMSE, 'MSE_tr': BGG1_trMSE, 'MAE_tr': BGG1_trMAE})
Compa_df = pd.concat([Compa_df, BGG1_df])
Compa_df
Out[158]:
Bagging model also performed well in training and validation sets.There seems to be overfitting in training set. We need to analyse further by hypertuning
Random forest¶
In [159]:
from sklearn.ensemble import RandomForestRegressor
In [160]:
RF1=RandomForestRegressor()
RF1.fit(X_train, y_train)
y_RF1_predtr= RF1.predict(X_train)
y_RF1_predvl= RF1.predict(X_val)
#Model score and Deduction for each Model in a DataFrame
RF1_trscore=r2_score(y_train,y_RF1_predtr)
RF1_trRMSE=np.sqrt(mean_squared_error(y_train, y_RF1_predtr))
RF1_trMSE=mean_squared_error(y_train, y_RF1_predtr)
RF1_trMAE=mean_absolute_error(y_train, y_RF1_predtr)
RF1_vlscore=r2_score(y_val,y_RF1_predvl)
RF1_vlRMSE=np.sqrt(mean_squared_error(y_val, y_RF1_predvl))
RF1_vlMSE=mean_squared_error(y_val, y_RF1_predvl)
RF1_vlMAE=mean_absolute_error(y_val, y_RF1_predvl)
RF1_df=pd.DataFrame({'Method':['RF1'],'Val Score':RF1_vlscore,'RMSE_vl': RF1_vlRMSE, 'MSE_vl':RF1_vlMSE, 'MAE_vl': RF1_vlMAE,'train Score':RF1_trscore,'RMSE_tr': RF1_trRMSE, 'MSE_tr': RF1_trMSE, 'MAE_tr': RF1_trMAE})
Compa_df = pd.concat([Compa_df, RF1_df])
Compa_df
Out[160]:
Random forest model has performed well in training and validation set. There is scope of further analysis on this model
Enseble models: in summary ensemble models have performed well on training and validation sets. These models will be selected for further analysis with hypertuning and feature selection¶
In [161]:
#feature importance
rf_imp_feature_1=pd.DataFrame(RF1.feature_importances_, columns = ["Imp"], index = X_val.columns)
rf_imp_feature_1.sort_values(by="Imp",ascending=False)
rf_imp_feature_1['Imp'] = rf_imp_feature_1['Imp'].map('{0:.5f}'.format)
rf_imp_feature_1=rf_imp_feature_1.sort_values(by="Imp",ascending=False)
rf_imp_feature_1.Imp=rf_imp_feature_1.Imp.astype("float")
rf_imp_feature_1[:30].plot.bar(figsize=(plotSizeX, plotSizeY))
#First 20 features have an importance of 90.5% and first 30 have importance of 95.15
print("First 20 feature importance:\t",(rf_imp_feature_1[:20].sum())*100)
print("First 30 feature importance:\t",(rf_imp_feature_1[:30].sum())*100)
Above are top 30 important features that account for 95% of variation in model. This need to be further analysed during hypertuning of the models for better scores
Model performance Summary:¶
Ensemble methods are performing better than linear models. Of all the ensemble models, Gradient boosting regressor is giving better R2 score. we identified top 30 features that are explaining the 95% variation in model(Random Forest). Will further hypertune the model to improve the model performance. Will further explore and evaluate the features while hyperturning the ensemble models
Building Function/Pipeline for models¶
In [162]:
rf_imp_feature_1[:30]
Out[162]:
In [163]:
from sklearn.pipeline import Pipeline
In [164]:
def result (model,pipe_model,X_train_set,y_train_set,X_val_set,y_val_set):
pipe_model.fit(X_train_set,y_train_set)
#predicting result over test data
y_train_predict= pipe_model.predict(X_train_set)
y_val_predict= pipe_model.predict(X_val_set)
trscore=r2_score(y_train_set,y_train_predict)
trRMSE=np.sqrt(mean_squared_error(y_train_set,y_train_predict))
trMSE=mean_squared_error(y_train_set,y_train_predict)
trMAE=mean_absolute_error(y_train_set,y_train_predict)
vlscore=r2_score(y_val,y_val_predict)
vlRMSE=np.sqrt(mean_squared_error(y_val,y_val_predict))
vlMSE=mean_squared_error(y_val,y_val_predict)
vlMAE=mean_absolute_error(y_val,y_val_predict)
result_df=pd.DataFrame({'Method':[model],'val score':vlscore,'RMSE_val':vlRMSE,'MSE_val':vlMSE,'MSE_vl': vlMSE,
'train Score':trscore,'RMSE_tr': trRMSE,'MSE_tr': trMSE, 'MAE_tr': trMAE})
return result_df
Above function will run the model and return the r2 score,rmse,mse of the model
In [165]:
#Creating empty dataframe to capture results
result_dff=pd.DataFrame()
pipe_LR = Pipeline([('LR', LinearRegression())])
result_dff=pd.concat([result_dff,result('LR',pipe_LR,X_train,y_train,X_val,y_val)])
pipe_knr = Pipeline([('KNNR', KNeighborsRegressor(n_neighbors=4,weights='distance'))])
result_dff=pd.concat([result_dff,result('KNNR',pipe_knr,X_train,y_train,X_val,y_val)])
pipe_DTR = Pipeline([('DTR', DecisionTreeRegressor())])
result_dff=pd.concat([result_dff,result('DTR',pipe_DTR,X_train,y_train,X_val,y_val)])
pipe_GBR = Pipeline([('GBR', GradientBoostingRegressor(n_estimators = 200, learning_rate = 0.1, random_state=22))])
result_dff=pd.concat([result_dff,result('GBR',pipe_GBR,X_train,y_train,X_val,y_val)])
pipe_BGR = Pipeline([('BGR', BaggingRegressor(n_estimators=50, oob_score= True,random_state=14))])
result_dff=pd.concat([result_dff,result('BGR',pipe_BGR,X_train,y_train,X_val,y_val)])
pipe_RFR = Pipeline([('RFR', RandomForestRegressor())])
result_dff=pd.concat([result_dff,result('RFR',pipe_RFR,X_train,y_train,X_val,y_val)])
result_dff
Out[165]:
Above sequence of steps with pipeline function will run all the models and compile the scores in result_dff dataframe. We can see that the above 2 steps are concise instead of running individual models and compiling the scores as earlier.
We can clearly see gradient boosting is giving better result in comparison with other ensemble methods. Also the score of 0.82 on training set indicates no overfitting of the model¶
In [166]:
#Storing results of initial data set - dff
result_ds1=result_dff.copy()
result_ds1
Out[166]:
FEATURE SELECTION (PCA)¶
Now, we will explore the possibility of features reduction using PCA
In [167]:
dff.shape
Out[167]:
In [168]:
dff.columns
Out[168]:
will drop the price column as it is the target variable
In [169]:
df_pca = dff.drop(['price'], axis = 1)
In [170]:
numerical_cols = df_pca.copy()
numerical_cols.shape
Out[170]:
In [171]:
# Let's first transform the entire X (independent variable data) to zscores.
# We will create the PCA dimensions on this distribution.
from scipy.stats import zscore
# As PCA for Independent columns of Numerical types, let's pass numerical_cols (16 numerical features)
numerical_cols = numerical_cols.apply(zscore)
cov_matrix = np.cov(numerical_cols.T)
print('Covariance Matrix \n%s', cov_matrix)
As we can see, near the value to 1, more the features related.
In [172]:
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)
print('Eigen Vectors \n%s', eigenvectors)
print('\n Eigen Values \n%s', eigenvalues)
In [173]:
# Let's Sort eigenvalues in descending order
# Make a set of (eigenvalue, eigenvector) pairs
eig_pairs = [(eigenvalues[index], eigenvectors[:,index]) for index in range(len(eigenvalues))]
# Sort the (eigenvalue, eigenvector) pairs from highest to lowest with respect to eigenvalue
eig_pairs.sort()
eig_pairs.reverse()
print(eig_pairs)
# Extract the descending ordered eigenvalues and eigenvectors
eigvalues_sorted = [eig_pairs[index][0] for index in range(len(eigenvalues))]
eigvectors_sorted = [eig_pairs[index][1] for index in range(len(eigenvalues))]
# Let's confirm our sorting worked, print out eigenvalues
print('Eigenvalues in descending order: \n%s' %eigvalues_sorted)
In [174]:
tot = sum(eigenvalues)
var_explained = [(i / tot) for i in sorted(eigenvalues, reverse=True)]
# an array of variance explained by each
# eigen vector... there will be 90 entries as there are 90 eigen vectors)
cum_var_exp = np.cumsum(var_explained)
# an array of cumulative variance. There will be 90 entries with 90 th entry cumulative reaching almost 100%
In [175]:
print(len(var_explained))
print((cum_var_exp))
From above table we conclude that 96% variance is contributed by about 72 features
In [176]:
plt.figure(figsize=(plotSizeX, plotSizeY))
plt.bar(range(0,90), np.array(var_explained), alpha = 0.5, align='center', label='individual explained variance')
plt.step(range(0,90), np.array(cum_var_exp), where= 'mid', label='cumulative explained variance')
plt.ylabel('Explained variance ratio')
plt.xlabel('Principal components')
plt.legend(loc = 'best')
plt.show()
72 dimensions covering 97% variance in the data. So we can reduce to 72 dimension space¶
Now will recall the ensemble models from our initial run to check the feature selection using featureimp from individual models
In [177]:
#Building fuction to return the feature importances for the model
predictors = [x for x in dff.columns if x not in ['price']]
def modelfit(alg, dxtrain, dytrain, printFeatureImportance=True):
#feature importance
alg.fit(dxtrain,dytrain)
alg_imp_feature_1=pd.DataFrame(alg.feature_importances_, columns = ["Imp"], index = predictors)
alg_imp_feature_1.sort_values(by="Imp",ascending=False)
alg_imp_feature_1['Imp'] = alg_imp_feature_1['Imp'].map('{0:.5f}'.format)
alg_imp_feature_1=alg_imp_feature_1.sort_values(by="Imp",ascending=False)
alg_imp_feature_1.Imp=alg_imp_feature_1.Imp.astype("float")
feat_30list=list(alg_imp_feature_1.index[:30])
if printFeatureImportance:
alg_imp_feature_1[:30].plot.bar(figsize=(plotSizeX, plotSizeY))
#First 20 features have an importance of 90.5% and first 30 have importance of 95.15
print("First 25 feature importance:\t",(alg_imp_feature_1[:25].sum())*100)
print("First 30 feature importance:\t",(alg_imp_feature_1[:30].sum())*100)
return feat_30list
Will run above function with ensemble models: Gradient boosting, Random forest, Bagging
In [178]:
#Gradient boost model
modelfit(GB1,X_train,y_train)
Out[178]:
The top 30 features are covering about 98% in gradient boosting model. This is very good coverage for just 30% of the variables
In [179]:
#Random Forest model
modelfit(RF1,X_train,y_train)
Out[179]:
The top 30 features are covering about 95% in random forest model
Now will extract the top 30 features from the above models
In [180]:
feat_list_GB1=modelfit(GB1,X_train,y_train, printFeatureImportance=False)
print(feat_list_GB1)
feat_list_RF1=modelfit(RF1,X_train,y_train, printFeatureImportance=False)
print(feat_list_RF1)
From the above 2 feature list, we will consolidate all the features
In [181]:
Key_feat=list(set(feat_list_GB1).union(feat_list_RF1))
print(len(Key_feat))
print(Key_feat)
From two models we have 33 importance features. We will freeze on the above 33 list and make another dataframe (along with 'price')
In [182]:
dff33=dff[['price','basement', 'City_Bellevue', 'coast_1', 'HouseLandRatio', 'City_Seattle', 'quality_10', 'quality_9', 'ceil_measure', 'City_Renton', 'City_Redmond', 'City_Federal Way', 'City_Mercer Island', 'yr_built', 'living_measure15', 'living_measure', 'City_Maple Valley', 'sight_3', 'total_area', 'City_Kirkland', 'sight_4', 'quality_6', 'quality_7', 'City_Sammamish', 'quality_8', 'City_Kent', 'quality_12', 'lot_measure', 'condition_3', 'furnished_1', 'City_Issaquah', 'quality_11', 'City_Medina', 'lot_measure15']].copy()
In [183]:
dff33.shape
Out[183]:
In [184]:
dff33.head()
Out[184]:
In [185]:
X3 = dff33.drop("price" , axis=1)
y3 = dff33["price"]
X3_train, X3_test, y3_train, y3_test = train_test_split(X3, y3, test_size=0.2, random_state=10)
X3_train, X3_val, y3_train, y3_val = train_test_split(X3_train, y3_train, test_size=0.2, random_state=10)
print(X3_train.shape)
print(X3_test.shape)
print(X3_val.shape)
Eventhough PCA is helping us to reduce dimensions upto about 60 dimensions, we can see that in our random forest model top 30 features are explaining the 95% variance in the regression and in gradient boosting model top 30 features are covering 98% varience.¶
Hence we conclude that we will use features selection by considering the feature importance fucntion in individual models. Thus we extracted 33 important features¶
HYPERTUNING with Gridsearch CV¶
In [186]:
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import accuracy_score
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import cross_val_score
Since we have better performance in gradient boosting model, we will hypertune the model for improving the score
Following are the parameters we tune for the gradient boosting model.
In [187]:
param_grid = {
'loss':['ls','lad','huber'],
'bootstrap': ['True','False'],
'max_depth': range(5,11,1),
'max_features': ['auto','sqrt'],
'learning_rate': [0.05,0.1,0.2,0.25],
'min_samples_leaf': [4,10,20],
'min_samples_split': [5,10,1000],
'n_estimators': [10,50,100,150,200],
'subsample':[0.8,1]
}
In [188]:
GBR_test=GradientBoostingRegressor(random_state=22)
First will tune each parameter separately
In [189]:
param_grid1 = {'n_estimators': range(50,401,50)}
In [190]:
grid_search1 = GridSearchCV(estimator = GBR_test, param_grid = param_grid1,
cv = 3, n_jobs = 2, verbose = 1)
In [191]:
grid_search1.fit(X_train,y_train)
grid_search1.best_params_
Out[191]:
In [192]:
grid_search1.best_params_, grid_search1.best_score_
Out[192]:
n_estimators of 400 is best in range 50 to 400. Will test same until 1000
In [193]:
param_grid2 = {'n_estimators': range(400,1001,200)}
GBR_test=GradientBoostingRegressor(random_state=22)
grid_search2 = GridSearchCV(estimator = GBR_test, param_grid = param_grid2,
cv = 3, n_jobs = 2, verbose = 1)
grid_search2.fit(X_train,y_train)
Out[193]:
In [194]:
grid_search2.cv_results_,grid_search2.best_params_, grid_search2.best_score_
Out[194]:
In [195]:
param_grid2 = {'n_estimators': range(1000,2000,300)}
GBR_test=GradientBoostingRegressor(random_state=22)
grid_search2 = GridSearchCV(estimator = GBR_test, param_grid = param_grid2,
cv = 5, n_jobs = 3, verbose = 1)
grid_search2.fit(X_train,y_train)
Out[195]:
In [196]:
grid_search2.best_params_, grid_search2.best_score_
Out[196]:
n_estimators of 1000 is giving best result in range 400 to 1000
In [197]:
param_grid3 = {
'learning_rate': [0.1,0.2],
'min_samples_leaf': [5,10,20],
'min_samples_split': [5,10,20],
'n_estimators': [500,1000],
}
In [198]:
GBR_test=GradientBoostingRegressor(random_state=22)
grid_search3 = GridSearchCV(estimator = GBR_test, param_grid = param_grid3,
cv = 5, n_jobs = 3, verbose = 1)
grid_search3.fit(X_train,y_train)
Out[198]:
In [199]:
grid_search3.best_params_, grid_search3.best_score_
Out[199]:
In combination of 4 parameters above values are giving best result. We can see n_estimators of 1000 is best again. Now, will change the ranges of other 3 parameters
In [200]:
param_grid4 = {
'learning_rate': [0.1,0.15],
'max_depth': [5,10],
'min_samples_leaf': [5,8],
'min_samples_split': [20,30],
'n_estimators': [1000],
}
In [201]:
GBR_test=GradientBoostingRegressor(random_state=22)
grid_search4 = GridSearchCV(estimator = GBR_test, param_grid = param_grid4,
cv = 5, n_jobs = 3, verbose = 1)
grid_search4.fit(X_train,y_train)
Out[201]:
In [202]:
grid_search4.best_params_, grid_search4.best_score_
Out[202]:
Now the score has reduced compared to earlier run
In [203]:
param_grid5 = {
'learning_rate': [0.1],
'max_depth': [5],
'min_samples_leaf': [8,10],
'min_samples_split': [30,40],
'n_estimators': [1000],
}
GBR_test=GradientBoostingRegressor(random_state=22)
grid_search5 = GridSearchCV(estimator = GBR_test, param_grid = param_grid5,
cv = 5, n_jobs = 2, verbose = 1)
grid_search5.fit(X_train,y_train)
Out[203]:
In [204]:
grid_search5.best_params_, grid_search5.best_score_
Out[204]:
Above score has improved from earlier runs
In [205]:
param_grid6 = {
'learning_rate': [0.1],
'max_depth': [5],
'min_samples_leaf': [8],
'min_samples_split': [40,50],
'n_estimators': [1000],
}
GBR_test=GradientBoostingRegressor(random_state=22)
grid_search6 = GridSearchCV(estimator = GBR_test, param_grid = param_grid6,
cv = 5, n_jobs = 2, verbose = 1)
grid_search6.fit(X_train,y_train)
Out[205]:
In [206]:
grid_search6.best_params_, grid_search6.best_score_
Out[206]:
There is very marginal improvment in score. We are getting best score at min_samples_split of 40 among 30,40,50.
Will tune the final set of parameters along with above finalized ones¶
In [207]:
param_grid7 = {
'loss':['ls','lad','huber'],
'max_features': ['auto','sqrt'],
'learning_rate': [0.1],
'max_depth': [5],
'min_samples_leaf': [8],
'min_samples_split': [40],
'n_estimators': [1000],
'subsample':[0.8,1]
}
GBR_test=GradientBoostingRegressor(random_state=22)
grid_search7 = GridSearchCV(estimator = GBR_test, param_grid = param_grid7,
cv = 5, n_jobs = 2, verbose = 1)
grid_search7.fit(X_train,y_train)
Out[207]:
In [208]:
grid_search7.best_params_, grid_search7.best_score_
Out[208]:
There is improvement in the score. will try one more iteration with changing other parameters
In [209]:
param_gridF = {
'loss':['huber'],
'max_features': ['sqrt'],
'learning_rate': [0.1,0.2],
'max_depth': [5,8],
'min_samples_leaf': [5],
'min_samples_split': [40,50],
'n_estimators': [1000],
'subsample':[1]
}
GBR_test=GradientBoostingRegressor(random_state=22)
grid_searchF = GridSearchCV(estimator = GBR_test, param_grid = param_gridF,
cv = 5, n_jobs = 2, verbose = 1)
grid_searchF.fit(X_train,y_train)
grid_searchF.best_params_,grid_searchF.best_score_
Out[209]:
'learning_rate': 0.1, 'loss': 'huber', 'max_depth': 5, 'max_features': 'sqrt', 'min_samples_leaf': 5, 'min_samples_split': 50, 'n_estimators': 1000, 'subsample': 1 </b>
Hypertuning using graph¶
In [210]:
min_samples_leafs = range(1, 15, 1)
train_results = []
val_results = []
for min_samples_leaf in min_samples_leafs:
GBR_test=GradientBoostingRegressor(
loss='huber',
learning_rate=0.1,
n_estimators=1000,
subsample=1.0,
min_samples_split=40,
min_samples_leaf=min_samples_leaf,
max_depth=5,
random_state=22,
alpha=0.9,
)
GBR_test.fit(X_train,y_train)
y_GBR_predtr= GBR_test.predict(X_train)
y_GBR_predvl= GBR_test.predict(X_val)
result_leafs_tr=r2_score(y_GBR_predtr,y_train)
train_results.append(result_leafs_tr)
result_leafs_vl=r2_score(y_GBR_predvl,y_val)
val_results.append(result_leafs_vl)
from matplotlib.legend_handler import HandlerLine2D
line1, = plt.plot(min_samples_leafs,train_results,"b", label='Train r2')
line2, = plt.plot(min_samples_leafs, val_results,"r", label='val r2')
plt.legend(handler_map={line1: HandlerLine2D(numpoints=2)})
plt.ylabel("r2 score")
plt.xlabel("min samples leaf")
plt.show()
From above, min_samples_leaf of 6 is giving best score
In [211]:
min_samples_splits = [10,15,30,50,100,500,700,1000]
train_results_spt = []
val_results_spt = []
for min_samples_split in min_samples_splits:
GBR_test=GradientBoostingRegressor(
loss='huber',
learning_rate=0.1,
n_estimators=1000,
subsample=1.0,
min_samples_split=min_samples_split,
min_samples_leaf=5,
max_depth=5,
random_state=22,
alpha=0.9,
)
GBR_test.fit(X_train,y_train)
y_GBR_predtr= GBR_test.predict(X_train)
y_GBR_predvl= GBR_test.predict(X_val)
result_spt_tr=r2_score(y_GBR_predtr,y_train)
train_results_spt.append(result_spt_tr)
result_spt_vl=r2_score(y_GBR_predvl,y_val)
val_results_spt.append(result_spt_vl)
from matplotlib.legend_handler import HandlerLine2D
line1, = plt.plot(min_samples_splits,train_results_spt,"b", label='Train R2')
line2, = plt.plot(min_samples_splits, val_results_spt,"r", label='Val R2')
plt.legend(handler_map={line1: HandlerLine2D(numpoints=2)})
plt.ylabel("R2 score")
plt.xlabel("min samples split")
plt.show()
From above, min_samples_splits of about 10 is giving best score. Will try expanding the range around 10
In [212]:
min_samples_splits = [10,15,20,30,40,50,60,70,80,90,100]
train_results_spt = []
val_results_spt = []
for min_samples_split in min_samples_splits:
GBR_test=GradientBoostingRegressor(
loss='huber',
learning_rate=0.1,
n_estimators=1000,
subsample=1.0,
min_samples_split=min_samples_split,
min_samples_leaf=5,
max_depth=5,
random_state=22,
alpha=0.9,
)
GBR_test.fit(X_train,y_train)
y_GBR_predtr= GBR_test.predict(X_train)
y_GBR_predvl= GBR_test.predict(X_val)
result_spt_tr=r2_score(y_GBR_predtr,y_train)
train_results_spt.append(result_spt_tr)
result_spt_vl=r2_score(y_GBR_predvl,y_val)
val_results_spt.append(result_spt_vl)
from matplotlib.legend_handler import HandlerLine2D
line1, = plt.plot(min_samples_splits,train_results_spt,"b", label='Train R2')
line2, = plt.plot(min_samples_splits, val_results_spt,"r", label='Val R2')
plt.legend(handler_map={line1: HandlerLine2D(numpoints=2)})
plt.ylabel("R2 score")
plt.xlabel("min samples split")
plt.show()
From above, min_samples_splits of about 10 is giving best score
In [213]:
min_samples_splits = [7,8,9,10,11,12,13,14,15,20]
train_results_spt = []
val_results_spt = []
for min_samples_split in min_samples_splits:
GBR_test=GradientBoostingRegressor(
loss='huber',
learning_rate=0.1,
n_estimators=1000,
subsample=1.0,
min_samples_split=min_samples_split,
min_samples_leaf=5,
max_depth=5,
random_state=22,
alpha=0.9,
)
GBR_test.fit(X_train,y_train)
y_GBR_predtr= GBR_test.predict(X_train)
y_GBR_predvl= GBR_test.predict(X_val)
result_spt_tr=r2_score(y_GBR_predtr,y_train)
train_results_spt.append(result_spt_tr)
result_spt_vl=r2_score(y_GBR_predvl,y_val)
val_results_spt.append(result_spt_vl)
from matplotlib.legend_handler import HandlerLine2D
line1, = plt.plot(min_samples_splits,train_results_spt,"b", label='Train R2')
line2, = plt.plot(min_samples_splits, val_results_spt,"r", label='Val R2')
plt.legend(handler_map={line1: HandlerLine2D(numpoints=2)})
plt.ylabel("R2 score")
plt.xlabel("min samples split")
plt.show()
From above, min_samples_splits of about 12 is giving best score
In [214]:
max_depths = range(3,11,1)
train_results_dpt = []
val_results_dpt = []
for max_depth in max_depths:
GBR_test=GradientBoostingRegressor(
loss='huber',
learning_rate=0.1,
n_estimators=1000,
subsample=1.0,
min_samples_split=10,
min_samples_leaf=6,
max_depth=max_depth,
random_state=22,
alpha=0.9,
)
GBR_test.fit(X_train,y_train)
y_GBR_predtr= GBR_test.predict(X_train)
y_GBR_predvl= GBR_test.predict(X_val)
result_dpt_tr=r2_score(y_GBR_predtr,y_train)
train_results_dpt.append(result_dpt_tr)
result_dpt_vl=r2_score(y_GBR_predvl,y_val)
val_results_dpt.append(result_dpt_vl)
from matplotlib.legend_handler import HandlerLine2D
line1, = plt.plot(max_depths,train_results_dpt,"b", label='Train R2')
line2, = plt.plot(max_depths, val_results_dpt,"r", label='Val R2')
plt.legend(handler_map={line1: HandlerLine2D(numpoints=2)})
plt.ylabel("R2 score")
plt.xlabel("max depth")
plt.show()
From above, max_depth of about 6 is giving best score for validation set and not overfitting of training set
In [215]:
estimators = range(100,1500,100)
train_results_est = []
val_results_est = []
for n_estimators in estimators:
GBR_test=GradientBoostingRegressor(
loss='huber',
learning_rate=0.1,
n_estimators=n_estimators,
subsample=1.0,
min_samples_split=30,
min_samples_leaf=6,
max_depth=9,
random_state=22,
alpha=0.9,
)
GBR_test.fit(X_train,y_train)
y_GBR_predtr= GBR_test.predict(X_train)
y_GBR_predvl= GBR_test.predict(X_val)
result_est_tr=r2_score(y_GBR_predtr,y_train)
train_results_est.append(result_est_tr)
result_est_vl=r2_score(y_GBR_predvl,y_val)
val_results_est.append(result_est_vl)
from matplotlib.legend_handler import HandlerLine2D
line1, = plt.plot(estimators,train_results_est,"b", label='Train R2')
line2, = plt.plot(estimators, val_results_est,"r", label='Val R2')
plt.legend(handler_map={line1: HandlerLine2D(numpoints=2)})
plt.ylabel("R2 score")
plt.xlabel("n_estimators")
plt.show()
From above, n_estimators of about 1000 is giving best score
In [217]:
param_gridF = {
'loss':['huber'],
'max_features': ['sqrt'],
'learning_rate': [0.1],
'max_depth': [6],
'min_samples_leaf': [6],
'min_samples_split': [12],
'n_estimators': [1000],
'subsample':[1]
}
GBR_test=GradientBoostingRegressor(random_state=22)
grid_searchF = GridSearchCV(estimator = GBR_test, param_grid = param_gridF,
cv = 5, n_jobs = 2, verbose = 1)
grid_searchF.fit(X_train,y_train)
grid_searchF.best_score_
Out[217]:
In [218]:
param_gridF = {
'loss':['huber'],
'max_features': ['sqrt'],
'learning_rate': [0.1],
'max_depth': [5],
'min_samples_leaf': [5],
'min_samples_split': [50],
'n_estimators': [1000],
'subsample':[1]
}
GBR_test=GradientBoostingRegressor(random_state=22)
grid_searchF = GridSearchCV(estimator = GBR_test, param_grid = param_gridF,
cv = 5, n_jobs = 2, verbose = 1)
grid_searchF.fit(X_train,y_train)
grid_searchF.best_score_,grid_searchF.best_params_
Out[218]:
We can conclude from above that gridsearch CV is giving better results compared to that of tuning done by graphical method of individual parameters
'learning_rate': 0.1, 'loss': 'huber', 'max_depth': 5, 'max_features': 'sqrt', 'min_samples_leaf': 5, 'min_samples_split': 50, 'n_estimators': 1000, 'subsample': 1 </b>
CONFIDENCE INTERVAL¶
In [219]:
GBR_bestparam=GradientBoostingRegressor(
loss='huber',
learning_rate=0.1,
n_estimators=1000,
subsample=1.0,
min_samples_split=50,
min_samples_leaf=5,
max_depth=5,
random_state=22,
alpha=0.9,
)
GBR_bestparam.fit(X_train,y_train)
y_GBRF_predtr= GBR_bestparam.predict(X_train)
y_GBRF_predvl= GBR_bestparam.predict(X_val)
y_GBRF_predts= GBR_bestparam.predict(X_test)
In [220]:
#Model score and Deduction for each Model in a DataFrame
GBRF_trscore=r2_score(y_train,y_GBRF_predtr)
GBRF_trRMSE=np.sqrt(mean_squared_error(y_train, y_GBRF_predtr))
GBRF_trMSE=mean_squared_error(y_train, y_GBRF_predtr)
GBRF_trMAE=mean_absolute_error(y_train, y_GBRF_predtr)
GBRF_vlscore=r2_score(y_val,y_GBRF_predvl)
GBRF_vlRMSE=np.sqrt(mean_squared_error(y_val, y_GBRF_predvl))
GBRF_vlMSE=mean_squared_error(y_val, y_GBRF_predvl)
GBRF_vlMAE=mean_absolute_error(y_val, y_GBRF_predvl)
GBRF_tsscore=r2_score(y_test,y_GBRF_predts)
GBRF_tsRMSE=np.sqrt(mean_squared_error(y_test, y_GBRF_predts))
GBRF_tsMSE=mean_squared_error(y_test, y_GBRF_predts)
GBRF_tsMAE=mean_absolute_error(y_test, y_GBRF_predts)
GBRF_df=pd.DataFrame({'Method':['GBRF'],'Val Score':GBRF_vlscore,'RMSE_vl': GBRF_vlRMSE, 'MSE_vl': GBRF_vlMSE,'train Score':GBRF_trscore,'RMSE_tr': GBRF_trRMSE, 'MSE_tr': GBRF_trMSE,'test Score':GBRF_tsscore,'RMSE_ts': GBRF_tsRMSE, 'MSE_ts': GBRF_tsMSE})
GBRF_df
Out[220]:
In [221]:
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
num_folds = 50
seed = 7
kfold = KFold(n_splits=num_folds, random_state=seed)
model = GradientBoostingRegressor(n_estimators = 200, learning_rate = 0.1, random_state=22)
results = cross_val_score(GBR_bestparam, X, y, cv=kfold)
print(results)
print("Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0))
In [222]:
from matplotlib import pyplot
# plot scores
pyplot.hist(results)
pyplot.show()
# confidence intervals
alpha = 0.95 # for 95% confidence
p = ((1.0-alpha)/2.0) * 100 # tail regions on right and left .25 on each side indicated by P value (border)
lower = max(0.0, np.percentile(results, p))
p = (alpha+((1.0-alpha)/2.0)) * 100
upper = min(1.0, np.percentile(results, p))
print('%.1f confidence interval %.1f%% and %.1f%%' % (alpha*100, lower*100, upper*100))
Dataset-1 Final summary:¶
Dataset-2¶
In [2]:
import geopandas as gpd
from shapely.geometry import Point, Polygon
#For current working directory
import os
cwd = os.getcwd()
In [224]:
## Need to add file USA ZipCodes_1.xlsx to current working directory to access this data
USAZip=pd.read_excel("USA ZipCodes_1.xlsx",sheet_name="Sheet8")
USAZip.head()
Out[224]:
In [239]:
house_df = pd.read_csv('innercity.csv')
In [240]:
house_df1=house_df.merge(USAZip,how='left',on='zipcode')
#house_df.drop_duplicates()
house_df.shape
Out[240]:
In [5]:
#Add the folder WA to your current working directory
usa = gpd.read_file(cwd+'\\WA\\WSDOT__City_Limits.shp')
usa.head()
gdf = gpd.GeoDataFrame(
house_df,geometry = [Point(xy) for xy in zip(house_df['long'], house_df['lat'])])
#We can now plot our ``GeoDataFrame``
ax=usa[usa.CityName.isin(house_df.City.unique())].plot(
color='white', edgecolor='black',figsize=(20,8))
plt.figure(figsize=(15,15))
gdf.plot(ax=ax, color='green', marker='o',markersize=0.1)
Out[5]:
In [241]:
#After analysis in p1 - Dropping 'cid','dayhours','basement','yr_built','yr_renovated','zipcode','lat','long','County','Type',
#'geometry','quality_group','month_year' columns.
cols=['cid','dayhours']
house_df_1=house_df.drop(cols, inplace = False, axis = 1)
The dataset worked earlier are giving r2 score on validation set in range 70%-75% with RMSE in range of 96000 to 155000. Trying with a different dataset to see if this could be improved further.
For analysis in this iteration categorizing coast, furnished and quality. As in previous version tranformed many features but not got desired result.
TREATING OUTLIERS¶
Removing data points which fall into below criteria:
- living_measure greater than 9000
- price greater than 4000000
- romm_bed greater than 10
- room_bath greater than 6
We have lost 20 records which is 0.09% of the data available. These records are extreme values for which we dont have much of data to provide their better estimate. Hence removing them.
In [242]:
house_df_2=house_df_1[(house_df['living_measure']<=9000) & (house_df_1['price']<=4000000) &
(house_df_1['room_bed']<=10) & (house_df_1['room_bath']<=6) ]
house_df_2.shape
Out[242]:
In [243]:
house_df_2.columns
Out[243]:
In [252]:
# Convert into dummies
house_df_final = pd.get_dummies(house_df_2, columns=['coast', 'quality', 'furnished'],drop_first=True)
In [253]:
house_df_final.columns
Out[253]:
In [254]:
house_df_final.shape
Out[254]:
In [268]:
#Final Data columns
house_df_final.columns
Out[268]:
Shows the Data Correlation between Attributes with Heatmap¶
In [256]:
#total_area is highly correlated with lot_measure, ceil_measure is highly correlated with living_measure
house_corr_2 = house_df_final.corr(method ='pearson')
house_corr_2.to_excel("house_corr_2.xls")
plt.figure(figsize=(35,20))
sns.heatmap(house_corr_2,cmap="coolwarm", annot=True,annot_kws={"size":9},fmt='.2')
Out[256]:
In [257]:
#creating a copy of the final dataframe
dff2=house_df_final.copy()
In [258]:
df_train, df_test = train_test_split(dff2, test_size=0.2, random_state=10)
df_train, df_val = train_test_split(df_train, test_size=0.2, random_state=10)
In [259]:
print(df_train.shape)
print(df_test.shape)
print(df_val.shape)
In [260]:
# Split the 'df_train' set into X and y
X_train2 = df_train.drop(['price'],axis=1)
y_train2 = df_train['price']
len_train=len(X_train2)
X_train2.shape
y_train2.head()
Out[260]:
In [261]:
# Split the 'df_val' set into X and y
X_val2 = df_val.drop(['price'],axis=1)
y_val2 = df_val['price']
len_val=len(X_val2)
X_val2.shape
y_val2.head()
Out[261]:
In [262]:
# Split the 'df_test' set into X and y
X_test2 = df_test.drop(['price'],axis=1)
y_test2 = df_test['price']
X_test2.shape
len_test=len(X_test2)
y_test2.head()
Out[262]:
Will use XGboost model apart from models that used earlier on dataset-1¶
Creating Dataframe for Results and Function to compute the scores for each model on its Train and Validation datasets¶
In [24]:
#Creating empty dataframe to capture results
result_dff=pd.DataFrame()
In [25]:
#Function to give results of the models for its train and validation dataset.
#as input it requries model name to display, algorithm, train indepedent variables, train dependent variable,
#validation indepedent variables, validation dependent variable.
def result (model,pipe_model,X_train_set,y_train_set,X_val_set,y_val_set):
pipe_model.fit(X_train_set,y_train_set)
#predicting result over test data
y_train_predict= pipe_model.predict(X_train_set)
y_val_predict= pipe_model.predict(X_val_set)
trscore=r2_score(y_train_set,y_train_predict)
trRMSE=np.sqrt(mean_squared_error(y_train_set,y_train_predict))
trMSE=mean_squared_error(y_train_set,y_train_predict)
trMAE=mean_absolute_error(y_train_set,y_train_predict)
vlscore=r2_score(y_val,y_val_predict)
vlRMSE=np.sqrt(mean_squared_error(y_val,y_val_predict))
vlMSE=mean_squared_error(y_val,y_val_predict)
vlMAE=mean_absolute_error(y_val,y_val_predict)
result_df=pd.DataFrame({'Method':[model],'val score':vlscore,'RMSE_val':vlRMSE,'MSE_val':vlMSE,'MAE_vl': vlMAE,
'train Score':trscore,'RMSE_tr': trRMSE,'MSE_tr': trMSE, 'MAE_tr': trMAE})
#Plot between actual and predicted values
plt.figure(figsize=(18,10))
sns.lineplot(range(len(y_val_set)),y_val_set,color='blue',linewidth=1.5)
sns.lineplot(range(len(y_val_set)),y_val_predict,color='hotpink',linewidth=.5)
plt.title('Actual and Predicted', fontsize=20) # Plot heading
plt.xlabel('Index', fontsize=10) # X-label
plt.ylabel('Values', fontsize=10) # Y-label
return result_df
LINEAR REGRESSION¶
In [26]:
#Starting with RFE first as there are many features
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import r2_score, mean_squared_error, mean_absolute_error
In [27]:
clf=LinearRegression()
pipe_lr = Pipeline([('LR', clf)])
result_dff=pd.concat([result_dff,result('Linear Reg',pipe_lr,X_train,y_train,X_val,y_val)])
result_dff
Out[27]:
In [28]:
#checking the magnitude of coefficients
predictors = X_train.columns
coef = pd.Series(clf.coef_,predictors).sort_values()
coef.plot(kind='bar', title='Model Coefficients',color='darkblue',figsize=(10,5))
Out[28]:
RIDGE REGRESSION¶
In [29]:
from sklearn.linear_model import Ridge
from sklearn.pipeline import Pipeline
from sklearn.metrics import r2_score, mean_squared_error, mean_absolute_error
from sklearn.preprocessing import StandardScaler
In [30]:
clf=Ridge()
pipe_ridge = Pipeline([('Ridge', clf)])
result_dff=pd.concat([result_dff,result('Ridge_Reg_1',pipe_ridge,X_train,y_train,X_val,y_val)])
result_dff
Out[30]:
In [31]:
#checking the magnitude of coefficients
predictors = X_train.columns
coef = pd.Series(clf.coef_,predictors).sort_values()
coef.plot(kind='bar', title='Model Coefficients',color='darkblue',figsize=(10,5))
Out[31]:
In [32]:
#Iteration 2
clf=Ridge(alpha=0.08)
pipe_ridge_1 = Pipeline([('Ridge',clf )])
result_dff=pd.concat([result_dff,result('Ridge_Reg_2',pipe_ridge_1,X_train,y_train,X_val,y_val)])
result_dff
Out[32]:
In [33]:
#checking the magnitude of coefficients
predictors = X_train.columns
coef = pd.Series(clf.coef_,predictors).sort_values()
coef.plot(kind='bar', title='Model Coefficients',color='darkblue',figsize=(10,5))
Out[33]:
LASSO REGRESSION¶
In [34]:
from sklearn.linear_model import Lasso
In [35]:
clf=Lasso(alpha=10, max_iter=1000)
pipe_lasso_1 = Pipeline([('Lasso',clf )])
result_dff=pd.concat([result_dff,result('Lasso_Reg_1',pipe_lasso_1,X_train,y_train,X_val,y_val)])
result_dff
Out[35]:
In [36]:
#checking the magnitude of coefficients
predictors = X_train.columns
coef = pd.Series(clf.coef_,predictors).sort_values(ascending=False)
coef
Out[36]:
KNN Regressor¶
In [37]:
from sklearn.neighbors import KNeighborsRegressor
pipe_knr = Pipeline([('KNNR', KNeighborsRegressor(n_neighbors=20,weights='distance'))])
result_dff=pd.concat([result_dff,result('KNN Reg',pipe_knr,X_train,y_train,X_val,y_val)])
result_dff
Out[37]:
Support Vector Regressor¶
In [38]:
#The model is not performing well at all.
#from sklearn.svm import SVR
#from sklearn.preprocessing import StandardScaler
#pipe_svr_1 = Pipeline([('scl', StandardScaler()),('SVR_1', SVR(kernel='rbf'))])
#result_dff=pd.concat([result_dff,result('SVR_1',pipe_svr_1,X_train_rfe,y_train,X_val_rfe,y_val)])
#result_dff
DECISION TREE¶
In [39]:
#Feature importance function
def feat_imp(model,X_data_set):
imp_feature_1=pd.DataFrame(model.feature_importances_, columns = ["Imp"], index = X_data_set.columns)
imp_feature_1=imp_feature_1.sort_values(by="Imp",ascending=False)
print(imp_feature_1)
#feature importance
plt.figure(figsize=(10,10))
imp_feature_1[:30].plot.bar(figsize=(15,5))
#First 20 and 30 feature importance sum
print("\nFirst 8 feature importance:\t",(imp_feature_1[:8].sum())*100)
print("\nFirst 12 feature importance:\t",(imp_feature_1[:12].sum())*100)
In [40]:
#Import library
from sklearn.tree import DecisionTreeRegressor
clf=DecisionTreeRegressor(random_state=1)
pipe_DT_1=Pipeline([('DT1',clf)])
result_dff=pd.concat([result_dff,result('DT1',pipe_DT_1,X_train,y_train,X_val,y_val)])
result_dff
Out[40]:
In [41]:
#Feature importance
feat_imp(clf,X_train)
RANDOM FOREST REGRESSOR¶
In [42]:
from sklearn.ensemble import RandomForestRegressor
In [43]:
clf=RandomForestRegressor(random_state=2)
pipe_RF_1=Pipeline([('RF1',clf)])
result_dff=pd.concat([result_dff,result('RF1',pipe_RF_1,X_train,y_train,X_val,y_val)])
result_dff
Out[43]:
In [44]:
#Feature importance
feat_imp(clf,X_train)
In [45]:
clf=RandomForestRegressor(n_estimators=50,max_depth=18,min_samples_leaf=10,random_state=3)
pipe_RF_2=Pipeline([('RF2',clf)])
result_dff=pd.concat([result_dff,result('RF2',pipe_RF_2,X_train,y_train,X_val,y_val)])
result_dff
Out[45]:
In [46]:
#Feature importance
feat_imp(clf,X_train)
Gradient Boost Regressor¶
In [47]:
from sklearn.ensemble import GradientBoostingRegressor
clf=GradientBoostingRegressor(random_state=4)
pipe_GB_1=Pipeline([('GB1',clf)])
result_dff=pd.concat([result_dff,result('GB1',pipe_GB_1,X_train,y_train,X_val,y_val)])
result_dff
Out[47]:
In [48]:
#Feature importance
feat_imp(clf,X_train)
In [49]:
clf=GradientBoostingRegressor(n_estimators=150,max_depth=5,random_state=5)
pipe_GB_2=Pipeline([('GB2',clf)])
result_dff=pd.concat([result_dff,result('GB2',pipe_GB_2,X_train,y_train,X_val,y_val)])
result_dff
Out[49]:
In [50]:
#Feature importance
feat_imp(clf,X_train)
XGBOOST REGRESSOR¶
In [51]:
from xgboost.sklearn import XGBRegressor
clf=XGBRegressor(objective='reg:squarederror',random_state=6)
pipe_XGB_1=Pipeline([('XGB1',clf)])
result_dff=pd.concat([result_dff,result('XGB1',pipe_XGB_1,X_train,y_train,X_val,y_val)])
result_dff
Out[51]:
In [52]:
#Feature importance
feat_imp(clf,X_train)
In [53]:
clf=XGBRegressor(n_estimators=150,max_depth=5,random_state=7)
pipe_XGB_2=Pipeline([('XGB2',clf)])
result_dff=pd.concat([result_dff,result('XGB2',pipe_XGB_2,X_train,y_train,X_val,y_val)])
result_dff
Out[53]:
In [54]:
#Feature importance
feat_imp(clf,X_train)
ADABOOST REGRESSOR¶
In [55]:
from sklearn.ensemble import AdaBoostRegressor
clf= AdaBoostRegressor(DecisionTreeRegressor(random_state=8))
pipe_ADAB_1=Pipeline([('ADAB1',clf)])
result_dff=pd.concat([result_dff,result('ADAB1',pipe_ADAB_1,X_train,y_train,X_val,y_val)])
result_dff
Out[55]:
In [56]:
#Feature importance
feat_imp(clf,X_train)
In [57]:
clf= AdaBoostRegressor(DecisionTreeRegressor(max_depth=20),n_estimators=250,learning_rate=0.005,random_state=9)
pipe_ADAB_2=Pipeline([('ADAB2',clf)])
result_dff=pd.concat([result_dff,result('ADAB2',pipe_ADAB_2,X_train,y_train,X_val,y_val)])
result_dff
Out[57]:
In [58]:
#Feature importance
feat_imp(clf,X_train)
BAGGING REGRESSION¶
In [59]:
from sklearn.ensemble import BaggingRegressor
clf= BaggingRegressor(random_state=10)
pipe_BAG_1=Pipeline([('BAG1',clf)])
result_dff=pd.concat([result_dff,result('BAG1',pipe_BAG_1,X_train,y_train,X_val,y_val)])
result_dff
Out[59]:
In [60]:
#Feature Importance
feature_importances = np.mean([ tree.feature_importances_ for tree in clf.estimators_], axis=0)
bg_imp_feature=pd.DataFrame(feature_importances, columns = ["Imp"],index=X_train.columns)
bg_imp_feature.sort_values(by="Imp",ascending=False)
Out[60]:
In [61]:
clf= BaggingRegressor(DecisionTreeRegressor(max_depth=12),n_estimators=250,random_state=11)
pipe_BAG_2=Pipeline([('BAG2',clf)])
result_dff=pd.concat([result_dff,result('BAG2',pipe_BAG_2,X_train,y_train,X_val,y_val)])
result_dff
Out[61]:
In [62]:
#Feature Importance
pd.options.display.float_format = '{:.5f}'.format
feature_importances = np.mean([ tree.feature_importances_ for tree in clf.estimators_], axis=0)
bg_imp_feature=pd.DataFrame(feature_importances, columns = ["Imp"],index=X_train.columns)
bg_imp_feature.sort_values(by="Imp",ascending=False)
Out[62]:
In [ ]:
Dataset-2 model performance Summary¶
We have used Linear Regression, Ridge and Lasso, KNN, Ensemble Techniques - Decision Trees, Random Forest, Bagging, AdaBoost, Gradient Boost and XGBoost - its gradient boost with regularization and its faster. R2 score on validation in range 70%-87% with RMSE in range 76000-107000. The model is showing better results.Lets hypertune to see if results could be improved further. Will use Random Forest, Gradient Boosting, XGBoost and AdaBoost hypertuning. Dropping features which are zero or very close to zero in all above 4 algos - quality_12, quality_3, quality_4.
Kindly refer Excel sheet to compare the results.
In [ ]:
#Dropping features
X_train_ht=X_train.drop(['quality_5', 'quality_3', 'quality_4'],1)
X_test_ht=X_test.drop(['quality_5', 'quality_3', 'quality_4'],1)
X_val_ht=X_val.drop(['quality_5', 'quality_3', 'quality_4'],1)
In [ ]:
skf = KFold(n_splits=5, random_state=12)
RANDOM FOREST HYPERTUNE¶
In [65]:
#Tuning of Random Forest
RF_ht = RandomForestRegressor()
params = {"n_estimators": np.arange(76,84,1),"max_depth": np.arange(16,20,1),
"max_features":np.arange(6,9,1),'min_samples_leaf': range(5, 8, 1),
'min_samples_split': range(18, 20, 1)}
RF_GV_1 = GridSearchCV(estimator = RF_ht, param_grid = params,cv=skf,verbose=1,return_train_score=True,n_jobs=2)
RF_GV_1.fit(X_train_ht,y_train)
Out[65]:
In [66]:
# results of grid search CV
RF_results = pd.DataFrame(RF_GV_1.cv_results_)
#parameters best value
best_score_rf = RF_GV_1.best_score_
best_rf = RF_GV_1.best_params_
best_rf
Out[66]:
In [67]:
rf_best = RandomForestRegressor(max_depth= 18, max_features= 8,n_estimators=80,min_samples_leaf=5,min_samples_split=18,
random_state=14)
result_dff=pd.concat([result_dff,result('RF_ht',rf_best,X_train_ht,y_train,X_val_ht,y_val)])
result_dff
Out[67]:
In [68]:
#Feature importance
feat_imp(rf_best,X_train_ht)
GRADIENT BOOST HYPERTUNE¶
In [69]:
GB_ht=GradientBoostingRegressor()
params = {"n_estimators": [138,142,1],"learning_rate":[0.08,0.09],"max_depth": np.arange(8, 11,1),
"max_features":np.arange(5,8,1),'min_samples_leaf': range(16, 21, 1)}
GB_GV_1 = GridSearchCV(estimator = GB_ht, param_grid = params,cv=skf,verbose=1,return_train_score=True,n_jobs=2)
GB_GV_1.fit(X_train_ht,y_train)
# results of grid search CV
GB_results = pd.DataFrame(GB_GV_1.cv_results_)
#parameters best value
best_score_rf = GB_GV_1.best_score_
best_gb = GB_GV_1.best_params_
best_gb
Out[69]:
In [70]:
gb_best = GradientBoostingRegressor(learning_rate= 0.09, n_estimators= 150,max_depth= 10,
max_features= 7,min_samples_leaf=19)
result_dff=pd.concat([result_dff,result('GB_ht',gb_best,X_train_ht,y_train,X_val_ht,y_val)])
result_dff
Out[70]:
In [71]:
#Feature importance
feat_imp(gb_best,X_train_ht)
ADABOOST HYPERTUNE¶
In [72]:
ADAB_ht=AdaBoostRegressor(DecisionTreeRegressor(max_depth=28))
params = {"n_estimators": [176,182,1],"learning_rate":[0.4,0.5,0.6],'loss':['linear','square']}
ADAB_GV_1 = GridSearchCV(estimator = ADAB_ht, param_grid = params,cv=skf,verbose=1,return_train_score=True,n_jobs=2)
ADAB_GV_1.fit(X_train_ht,y_train)
Out[72]:
In [73]:
# results of grid search CV
ADAB_results = pd.DataFrame(ADAB_GV_1.cv_results_)
#parameters best value
best_score_rf = ADAB_GV_1.best_score_
best_adab = ADAB_GV_1.best_params_
best_adab
Out[73]:
In [74]:
adab_best = AdaBoostRegressor(DecisionTreeRegressor(max_depth=28),n_estimators=180,learning_rate=0.5,loss='linear',
random_state=15)
result_dff=pd.concat([result_dff,result('ADAB_ht',adab_best,X_train_ht,y_train,X_val_ht,y_val)])
result_dff
Out[74]:
In [75]:
#Feature importance
feat_imp(adab_best,X_train_ht)
XGBoost Regressor¶
In [76]:
#Regularization using GridSearchCV - 1st Iteration
XGB_ht_1=XGBRegressor(objective='reg:squarederror')
params1 = {
"colsample_bytree": [i/100.0 for i in range(66,74,2)],
"learning_rate": [0.2,0.22,0.24],
"n_estimators": [185,188,1],
"subsample": [i/100.0 for i in range(62,68,1)]
}
XGB_GV_1 = GridSearchCV(estimator = XGB_ht_1, param_grid = params1,
cv=skf,
verbose = 1,
return_train_score=True,n_jobs=2)
XGB_GV_1.fit(X_train_ht,y_train)
Out[76]:
In [77]:
# results of grid search CV
XGB_results_1 = pd.DataFrame(XGB_GV_1.cv_results_)
#parameters best value
best_score_xgb_1 = XGB_GV_1.best_score_
best_xgb_1 = XGB_GV_1.best_params_
best_xgb_1
Out[77]:
In [78]:
#Choosing best parameter from 1st Iteration
xgb_best_1 = XGBRegressor(colsample_bytree=0.7,learning_rate=0.22,n_estimators=186,subsample=0.65,objective='reg:squarederror',
random_state=16)
result_dff=pd.concat([result_dff,result('xgb_1_ht',xgb_best_1,X_train_ht,y_train,X_val_ht,y_val)])
result_dff
Out[78]:
In [79]:
#Feature importance
feat_imp(xgb_best_1,X_train_ht)
In [80]:
#Regularization using GridSearchCV - 2nd Iteration
params2 = {
'min_child_weight':[6,7,8,9,10],"max_depth": [3,4,5],
}
xgb_best_2 = GridSearchCV(estimator = xgb_best_1, param_grid = params2,
cv=skf,
verbose = 1,
return_train_score=True,n_jobs=2)
xgb_best_2.fit(X_train_ht, y_train)
# results of grid search CV
XGB_results_2 = pd.DataFrame(xgb_best_2.cv_results_)
XGB_results_2
#parameters best value
best_score_xgb_2 = xgb_best_2.best_score_
best_xgb_2 = xgb_best_2.best_params_
best_xgb_2
Out[80]:
In [81]:
#Choosing best parameter from 2nd Iteration
xgb_best_2 = XGBRegressor(colsample_bytree=0.7,learning_rate=0.22,n_estimators=186,subsample=0.65,objective='reg:squarederror',
random_state=17,max_depth=4,min_child_weight=8)
result_dff=pd.concat([result_dff,result('xgb_2_ht',xgb_best_2,X_train_ht,y_train,X_val_ht,y_val)])
result_dff
Out[81]:
In [82]:
#Feature importance
feat_imp(xgb_best_2,X_train_ht)
In [83]:
#Regularization using GridSearchCV - 3rd Iteration
params3 = {
'gamma':[i/1.0 for i in range(50,55,1)]
}
xgb_best_3 = GridSearchCV(estimator = xgb_best_2, param_grid = params3,
cv=skf,
verbose = 1,
return_train_score=True)
xgb_best_3.fit(X_train_ht, y_train)
# results of grid search CV
XGB_results_3 = pd.DataFrame(xgb_best_3.cv_results_)
XGB_results_3
#parameters best value
best_score_xgb_3 = xgb_best_3.best_score_
best_xgb_3 = xgb_best_3.best_params_
best_xgb_3
Out[83]:
In [84]:
#Choosing best parameter from 3rd Iteration
xgb_best_3 = XGBRegressor(colsample_bytree=0.7,learning_rate=0.22,n_estimators=186,subsample=0.65,objective='reg:squarederror',
random_state=18,max_depth=4,min_child_weight=8,reg_lambda=52)
result_dff=pd.concat([result_dff,result('xgb_3_ht',xgb_best_3,X_train_ht,y_train,X_val_ht,y_val)])
result_dff
Out[84]:
In [85]:
#Feature importance
feat_imp(xgb_best_3,X_train_ht)
We have executed many models and post comparing results we hyper tuned four models. All models are working well with R2 score greater than 86% RMSE is below 132600.
But best of of all is Xtreme Gradient boost - which is enhanced version of gradient boost. It includes regularisation and is faster too. Its giving R2 score of around 89.5% with RMSE of around 109000.
Moving forward this model can be improved further as dont have much data for very high priced houses. So when more data comes in we can revisit our model and make mecessary changes to accommodate more variation in data to deliver better results, maybe try to decrease RMSE.
Finally lets run our model on test data, which we havent used till now and see how it performs.
Executing xgb_3_ht on test data set¶
In [86]:
result_dff=pd.concat([result_dff,result('xgb_test',xgb_best_3,X_test_ht,y_test,X_val_ht,y_val)])
result_dff
Out[86]:
In [87]:
#Feature importance
feat_imp(xgb_best_3,X_test_ht)
CALCULATING CONFIDENCE INTERVAL ON THE FINAL SELECTED MODEL at 95% ALPHA¶
In [88]:
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
num_folds = 200
seed = 7
kfold = KFold(n_splits=num_folds, random_state=seed)
results = cross_val_score(xgb_best_3, X_test_ht, y_test, cv=kfold)
print(results)
print("Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0))
In [89]:
from matplotlib import pyplot
# plot scores
pyplot.hist(results)
pyplot.show()
# confidence intervals
alpha = 0.95 # for 95% confidence
p = ((1.0-alpha)/2.0) * 100 # tail regions on right and left .25 on each side indicated by P value (border)
lower = max(0.0, np.percentile(results, p))
p = (alpha+((1.0-alpha)/2.0)) * 100
upper = min(1.0, np.percentile(results, p))
print('%.1f confidence interval %.1f%% and %.1f%%' % (alpha*100, lower*100, upper*100))
print('Average accuracy result on test data is %.3f%%:' % (np.mean(results)*100))
In [92]:
sns.set(style="darkgrid", color_codes=True)
with sns.axes_style("white"):
sns.jointplot(x=y_val, y=xgb_best_3.predict(X_val_ht), kind="reg", color="k")
plt.title('Actual and Predicted', fontsize=20) # Plot heading
plt.xlabel('Actual', fontsize=10) # X-label
plt.ylabel('Predicted', fontsize=10)
plt.tight_layout()
Dataset-2 Final summary¶
Finally we have the result, our final selected model is performing well on the test data R2 score of around 87.0% with RMSE of around 120000. </i>
Most important feature for pricing is furnished.The furnished house is priced higher.
Some other important features that affect price the most are living measure, latitude, above average quality of house and coastal house. So, one needs to thoroughly introspect its property on parameters suggested and list its price accordingly, similarly if one wants buy house - needs to check the features suggested above in house and calculate the predicted price. The same can than be compared to listed price.
Dataset-1 Final summary:¶
CONCLUSION:¶
We have build different models on 2 datasets. The performance (score and 95% confidence interval scores) of the model build on dataset-1 is better than dataset-2 as the 95% confidence interval of dataset-1 is very narrow compared to that of dataset-2. Even though the score of dataset-2 model is higher, the model has very vast range of performance scores.
The top key features to consider for pricing a property are:'furnished_1', 'yr_built', 'living_measure','quality_8', 'lot_measure15', 'quality_9', 'ceil_measure', 'total_area'. These are almost similar in both the models
So, one needs to thoroughly introspect its property on parameters suggested and list its price accordingly, similarly if one wants buy house - needs to check the features suggested above in house and calculate the predicted price. The same can than be compared to listed price.
For further improvization, the datasets can be made by treating outliers in different ways and hypertuning the ensemble models. Making polynomial features and improvising the model performance can also be explored further.
Pickle file Creation¶
First we will define the function for data-preprocessing that is required to run through the model. Then we will recall the same for predicting the price(target) of the property.
The pickle file is created as per the steps followed for dataset-2.
In [9]:
#Defining Funcation to process all required steps as done in model
def model(data):
import pandas as pd
import numpy as np
X_test = pd.read_excel(data)
#Removing outliers
X_test_1=X_test[(X_test['living_measure']<=9000) & (X_test['price']<=4000000) &
(X_test['room_bed']<=10) & (X_test['room_bath']<=6)]
cols=['cid','dayhours']
X_test_1=X_test.drop(cols, inplace = False, axis = 1)
#columns to be converted to category
categ=['coast', 'furnished','quality']
#X_test_2=X_test_1[categ].astype('category')
# Concatenate X_test_dummy_1 variables with X_test_2
#X_test_final = pd.concat([X_test_1, X_test_2], axis=1)
X_test_final=X_test_1.copy()
for i in range(1,2):
X_test_final['coast_'+str(i)]=0
X_test_final['furnished_'+str(i)]=0
for i in range(1,14):
X_test_final['quality_'+str(i)]=0
for i in range(1,2):
if ((X_test_final['coast']==i).bool()):
X_test_final['coast_'+str(i)]=1
for i in range(1,2):
if ((X_test_final['furnished']==i).bool()):
X_test_final['furnished_'+str(i)]=1
for i in range(1,14):
if ((X_test_final['quality']==i).bool()):
X_test_final['quality_'+str(i)]=1
X_test_final=X_test_final.drop([ 'quality_3', 'quality_4', 'quality_1', 'quality_2', 'quality_5','price'],1)
# Drop categorical variable columns
X_test_final = X_test_final.drop(X_test_final[categ], axis=1)
return X_test_final
Test run on pickle file:¶
In [ ]:
import pickle
with open('model_pickle','wb') as f:
pickle.dump(xgb_best_3,f)
In [11]:
with open('model_pickle','rb') as f:
mp=pickle.load(f)
In [14]:
X_test=model('innercity.xlsx')
mp.predict(X_test)
#X_test.columns
Out[14]:
We can see that with the given parameters, pickle file has run through the model and given predicted price of the property¶
In [ ]: